[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:21 dist-test-slave-b69v krb5kdc[5595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:21 dist-test-slave-b69v krb5kdc[5595](info): set up 2 sockets
Sep 29 15:57:21 dist-test-slave-b69v krb5kdc[5595](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 15:57:22.856549 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:22 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:23.044970 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38245
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:23.446702 5608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:23.447198 5608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:23.447377 5608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:23.455590 5608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:23.455792 5608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:23.455924 5608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:23.456038 5608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:23.456187 5608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:23.467751 5608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:23.469662 5608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:23.471396 5608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:23.487150 5613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:23.487211 5614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:23.487211 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:23.487934 5608 server_base.cc:1034] running on GCE node
I20260929 15:57:23.489041 5608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:23.490962 5608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:23.492179 5608 hybrid_clock.cc:648] HybridClock initialized: now 1790697443492105 us; error 64 us; skew 500 ppm
I20260929 15:57:23.492614 5608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:23 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:23.496556 5608 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:23.498469 5608 webserver.cc:460] Webserver started at http://127.5.117.62:43449/ using document root <none> and password file <none>
I20260929 15:57:23.499292 5608 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:23.499405 5608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:23.499696 5608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:23.503644 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "f38b25eb7222488d85a472ac493a2104"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-b69v"
server_key: "be5c65e6330ae64e23614ce48809ce78"
server_key_iv: "4d8d541a5ff273ee49014281bd69846e"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.504539 5608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "f38b25eb7222488d85a472ac493a2104"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-b69v"
server_key: "be5c65e6330ae64e23614ce48809ce78"
server_key_iv: "4d8d541a5ff273ee49014281bd69846e"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.510596 5608 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:23.515172 5623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:23.516776 5608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:57:23.516983 5608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "f38b25eb7222488d85a472ac493a2104"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-b69v"
server_key: "be5c65e6330ae64e23614ce48809ce78"
server_key_iv: "4d8d541a5ff273ee49014281bd69846e"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.517167 5608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:23.542168 5608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:23.547240 5608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:23.547494 5608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:23.569742 5674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:38245 every 8 connection(s)
I20260929 15:57:23.569870 5608 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:38245
I20260929 15:57:23.571893 5608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:23.576772 5675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.578506 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 5608
I20260929 15:57:23.578899 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:23.581087 5588 external_mini_cluster.cc:1428] Setting key 94764fcc1920cc64094b66cea223e452
I20260929 15:57:23.590432 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104: Bootstrap starting.
I20260929 15:57:23.594363 5675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.595678 5675 log.cc:824] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:23.600607 5675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104: No bootstrap required, opened a new log
I20260929 15:57:23.609943 5675 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } }
I20260929 15:57:23.610500 5675 raft_consensus.cc:374] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.610635 5675 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38b25eb7222488d85a472ac493a2104, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.611409 5675 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } }
I20260929 15:57:23.611713 5675 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:23.611876 5675 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:23.612107 5675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 15:57:23 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:23.615145 5675 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } }
I20260929 15:57:23.615803 5675 leader_election.cc:302] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38b25eb7222488d85a472ac493a2104; no voters:
I20260929 15:57:23.616853 5675 leader_election.cc:288] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:23.617046 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:23.618872 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [term 1 LEADER]: Becoming Leader. State: Replica: f38b25eb7222488d85a472ac493a2104, State: Running, Role: LEADER
I20260929 15:57:23.619611 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } }
I20260929 15:57:23.619953 5675 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:23.629173 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f38b25eb7222488d85a472ac493a2104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } } }
I20260929 15:57:23.629658 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:23.630102 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38b25eb7222488d85a472ac493a2104. Latest consensus state: current_term: 1 leader_uuid: "f38b25eb7222488d85a472ac493a2104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38b25eb7222488d85a472ac493a2104" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38245 } } }
I20260929 15:57:23.630417 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:23.631703 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.585806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.5.117.62:38245)
0929 15:57:23.587029 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.587049 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.588323 (+ 1274us) server_negotiation.cc:407] Connection header received
0929 15:57:23.598830 (+ 10507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.598855 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.599321 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.599847 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:23.603532 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.605116 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.606881 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.607947 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.608292 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.615857 (+ 7565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:23.615868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:23.615887 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:23.615945 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:23.620508 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.621683 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.621696 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.621705 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.621984 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.622759 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.622762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.622764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.623376 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:23.623725 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.627395 (+ 3670us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.627990 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":489,"threads_started":1}
I20260929 15:57:23.646139 5689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:23.651762 5689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:23.662145 5689 catalog_manager.cc:1347] Generated new cluster ID: e4ebc7d9963b4a1ebe0f18b1aaf2e5e6
I20260929 15:57:23.662262 5689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:23.683499 5689 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:23.685225 5689 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:23.694948 5689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f38b25eb7222488d85a472ac493a2104: Generated new TSK 0
I20260929 15:57:23.696341 5689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:23.760468 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:24.136813 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:24.137167 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:24.137348 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:24.144446 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:24.144582 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:24.144764 5701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:24.154502 5701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:24.155862 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:24.157330 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:24.171408 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:24.171383 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:24.172425 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:24.172540 5701 server_base.cc:1034] running on GCE node
I20260929 15:57:24.173202 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:24.174139 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:24.175361 5701 hybrid_clock.cc:648] HybridClock initialized: now 1790697444175318 us; error 55 us; skew 500 ppm
I20260929 15:57:24.175676 5701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:24 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:24.179026 5701 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:24.180509 5701 webserver.cc:460] Webserver started at http://127.5.117.1:39791/ using document root <none> and password file <none>
I20260929 15:57:24.181144 5701 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:24.181244 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:24.181519 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:24.184476 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "3ec2111967cf4f73ae25d8bb86638c25"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "fa79f0ae96261e731d599c627a9388ef"
server_key_iv: "07302fca29bc49f927e6b997f67c40ec"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.185262 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec2111967cf4f73ae25d8bb86638c25"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "fa79f0ae96261e731d599c627a9388ef"
server_key_iv: "07302fca29bc49f927e6b997f67c40ec"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.190495 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:24.193892 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.195439 5701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:24.195616 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "3ec2111967cf4f73ae25d8bb86638c25"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "fa79f0ae96261e731d599c627a9388ef"
server_key_iv: "07302fca29bc49f927e6b997f67c40ec"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.195808 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:24.214954 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:24.219404 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:24.219697 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:24.220757 5701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:24.222258 5701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:24.222334 5701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.222488 5701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:24.222568 5701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.257637 5701 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:41239
I20260929 15:57:24.257668 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:41239 every 8 connection(s)
I20260929 15:57:24.258909 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:24.268898 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 5701
I20260929 15:57:24.269135 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:24.269747 5588 external_mini_cluster.cc:1428] Setting key d053da84bc0c34593773b64850b9a2c5
Sep 29 15:57:24 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:24.279929 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.261611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:32993 (local address 127.5.117.62:38245)
0929 15:57:24.262146 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.262152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.263028 (+ 876us) server_negotiation.cc:407] Connection header received
0929 15:57:24.264986 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.264992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.265082 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.265202 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.266640 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.267474 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.269121 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.269564 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.269693 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.273853 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.273858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.273861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.273899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.276596 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.277471 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.277475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.277477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.277620 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.278022 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.278029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.278034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.278400 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.278553 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.279166 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.279410 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":305,"threads_started":1}
I20260929 15:57:24.280476 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.262063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:38245 (local address 127.5.117.1:32993)
0929 15:57:24.262866 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:24.262915 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:24.264703 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:24.265430 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:24.265438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:24.265911 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:24.266471 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.266488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.267615 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.267622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.268979 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.268996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.269777 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.269785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.270013 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.270672 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:24.270708 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:24.273627 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:24.276719 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:24.276729 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:24.276745 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:24.277354 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:24.277734 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:24.277738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:24.277740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:24.277910 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:24.278516 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:24.278521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:24.278809 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 15:57:24.279025 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 15:57:24.279369 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":293,"threads_started":1}
I20260929 15:57:24.282155 5829 heartbeater.cc:347] Connected to a master server at 127.5.117.62:38245
I20260929 15:57:24.282477 5829 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:24.283396 5829 heartbeater.cc:511] Master 127.5.117.62:38245 requested a full tablet report, sending...
I20260929 15:57:24.285797 5640 ts_manager.cc:194] Registered new tserver with Master: 3ec2111967cf4f73ae25d8bb86638c25 (127.5.117.1:41239)
I20260929 15:57:24.287292 5640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:32993
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:24.311340 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:24.709270 5835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:24.709692 5835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:24.709874 5835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:24.718335 5835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:24.718492 5835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:24.718706 5835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:24.729763 5835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:24.731371 5835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:24.733218 5835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:24.747730 5843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:24.747803 5840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:24.747776 5841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:24.748835 5835 server_base.cc:1034] running on GCE node
I20260929 15:57:24.749382 5835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:24.750344 5835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:24.751583 5835 hybrid_clock.cc:648] HybridClock initialized: now 1790697444751532 us; error 69 us; skew 500 ppm
I20260929 15:57:24.751906 5835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:24 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:24.755553 5835 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:24.757275 5835 webserver.cc:460] Webserver started at http://127.5.117.2:33629/ using document root <none> and password file <none>
I20260929 15:57:24.758033 5835 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:24.758184 5835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:24.758553 5835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:24.762552 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "1d0af03d8b7b42008242b485ee46bc8d"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "06e09edead0a72652a7192e8e5a080a6"
server_key_iv: "391906bffc9fda10dea1fadfa505d566"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.763485 5835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "1d0af03d8b7b42008242b485ee46bc8d"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "06e09edead0a72652a7192e8e5a080a6"
server_key_iv: "391906bffc9fda10dea1fadfa505d566"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.769778 5835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:57:24.773674 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.775100 5835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:24.775290 5835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "1d0af03d8b7b42008242b485ee46bc8d"
format_stamp: "Formatted at 2026-09-29 15:57:24 on dist-test-slave-b69v"
server_key: "06e09edead0a72652a7192e8e5a080a6"
server_key_iv: "391906bffc9fda10dea1fadfa505d566"
server_key_version: "encryptionkey@0"
I20260929 15:57:24.775524 5835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:24.804853 5835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:24.809666 5835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:24.809942 5835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:24.810935 5835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:24.812477 5835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:24.812559 5835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.812664 5835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:24.812712 5835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:24.849730 5835 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:34801
I20260929 15:57:24.849763 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:34801 every 8 connection(s)
I20260929 15:57:24.851284 5835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:24.853170 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 5835
I20260929 15:57:24.853430 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:24.854038 5588 external_mini_cluster.cc:1428] Setting key 2ccab4f48720584f005bb8c2cf8aaa8c
Sep 29 15:57:24 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697444, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:57:24.874111 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.854813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:43307 (local address 127.5.117.62:38245)
0929 15:57:24.855392 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.855399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.856523 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 15:57:24.858636 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.858642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.858736 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.858860 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.860634 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.861442 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.863135 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.863595 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.863752 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.867781 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.867786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.867789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.867821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.870628 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.871383 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.871387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.871390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.871518 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.872153 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.872162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.872164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.872453 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.872613 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.873424 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.873670 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
I20260929 15:57:24.875402 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.855518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:38245 (local address 127.5.117.2:43307)
0929 15:57:24.856359 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:24.856398 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:24.858341 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:24.859119 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:24.859128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:24.859703 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:24.860411 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.860430 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.861599 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.861606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.862974 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.862986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.863854 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.863862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.864194 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.864973 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:24.864997 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:24.867552 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:24.870771 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:24.870780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:24.870798 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:24.871235 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:24.871660 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:24.871665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:24.871669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:24.871993 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:24.872601 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:24.872609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:24.873033 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 15:57:24.873319 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 15:57:24.873976 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":18304,"thread_start_us":494,"threads_started":1}
I20260929 15:57:24.877594 5963 heartbeater.cc:347] Connected to a master server at 127.5.117.62:38245
I20260929 15:57:24.877954 5963 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:24.878799 5963 heartbeater.cc:511] Master 127.5.117.62:38245 requested a full tablet report, sending...
I20260929 15:57:24.880734 5640 ts_manager.cc:194] Registered new tserver with Master: 1d0af03d8b7b42008242b485ee46bc8d (127.5.117.2:34801)
I20260929 15:57:24.881683 5640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:43307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:24.904160 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:25.286059 5969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:25.286310 5969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:25.286543 5969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:57:25.290841 5829 heartbeater.cc:503] Master 127.5.117.62:38245 was elected leader, sending a full tablet report...
W20260929 15:57:25.293772 5969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:25.293890 5969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:25.294101 5969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:25.303514 5969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:38245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:25.304822 5969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:25.306103 5969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:25.319550 5974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:25.319648 5977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:25.319643 5975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:25.320820 5969 server_base.cc:1034] running on GCE node
I20260929 15:57:25.321355 5969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:25.322353 5969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:25.323570 5969 hybrid_clock.cc:648] HybridClock initialized: now 1790697445323534 us; error 48 us; skew 500 ppm
I20260929 15:57:25.323889 5969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:25.327241 5969 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:25.328742 5969 webserver.cc:460] Webserver started at http://127.5.117.3:36089/ using document root <none> and password file <none>
I20260929 15:57:25.329429 5969 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:25.329533 5969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:25.329799 5969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:25.332707 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "33370187c5af4553af4960daacf6aff5"
format_stamp: "Formatted at 2026-09-29 15:57:25 on dist-test-slave-b69v"
server_key: "97f1fa453f6d943078e39258b1766996"
server_key_iv: "00747bbb7bf3c6d474f89ac1a4c35319"
server_key_version: "encryptionkey@0"
I20260929 15:57:25.333601 5969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "33370187c5af4553af4960daacf6aff5"
format_stamp: "Formatted at 2026-09-29 15:57:25 on dist-test-slave-b69v"
server_key: "97f1fa453f6d943078e39258b1766996"
server_key_iv: "00747bbb7bf3c6d474f89ac1a4c35319"
server_key_version: "encryptionkey@0"
I20260929 15:57:25.338850 5969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:25.342141 5984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:25.343458 5969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:57:25.343585 5969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "33370187c5af4553af4960daacf6aff5"
format_stamp: "Formatted at 2026-09-29 15:57:25 on dist-test-slave-b69v"
server_key: "97f1fa453f6d943078e39258b1766996"
server_key_iv: "00747bbb7bf3c6d474f89ac1a4c35319"
server_key_version: "encryptionkey@0"
I20260929 15:57:25.343737 5969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:25.361521 5969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:25.365331 5969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:25.365562 5969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:25.366425 5969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:25.367846 5969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:25.367954 5969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:25.368113 5969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:25.368192 5969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 15:57:25.405652 5969 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:46017
I20260929 15:57:25.405728 6096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:46017 every 8 connection(s)
I20260929 15:57:25.406893 5969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:25.416718 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 5969
I20260929 15:57:25.416937 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:25.417418 5588 external_mini_cluster.cc:1428] Setting key bddbd06f1547be1a52c9b8729b5c43bc
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:25.426919 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.409626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:35827 (local address 127.5.117.62:38245)
0929 15:57:25.410120 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.410125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.410933 (+ 808us) server_negotiation.cc:407] Connection header received
0929 15:57:25.412881 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.412887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.413005 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.413135 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.414616 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.415495 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.417210 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.417654 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.417788 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.421824 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.421829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.421832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.421866 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.424485 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.425066 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.425070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.425072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.425183 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.425541 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.425544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.425546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.425802 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.425918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.426469 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.426631 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260929 15:57:25.427697 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.410051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:38245 (local address 127.5.117.3:35827)
0929 15:57:25.410769 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:25.410804 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:25.412591 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:25.413369 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:25.413378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:25.413888 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:25.414453 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:25.414469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.415646 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.415656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:25.417075 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:25.417086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.417874 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.417881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:25.418115 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.418789 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:25.418816 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:25.421581 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:25.424592 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:25.424598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:25.424608 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:25.424974 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:25.425285 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:25.425288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:25.425290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:25.425452 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:25.425905 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:25.425910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:25.426167 (+ 257us) client_negotiation.cc:769] Sending connection context
0929 15:57:25.426369 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 15:57:25.426697 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":309,"threads_started":1}
I20260929 15:57:25.428982 6097 heartbeater.cc:347] Connected to a master server at 127.5.117.62:38245
I20260929 15:57:25.429234 6097 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:25.429848 6097 heartbeater.cc:511] Master 127.5.117.62:38245 requested a full tablet report, sending...
I20260929 15:57:25.431159 5640 ts_manager.cc:194] Registered new tserver with Master: 33370187c5af4553af4960daacf6aff5 (127.5.117.3:46017)
I20260929 15:57:25.432122 5640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:35827
I20260929 15:57:25.434180 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:25.480945 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.465883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.5.117.1:41239)
0929 15:57:25.467049 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.467056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.467084 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:57:25.467148 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.467155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.467348 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.467472 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.468961 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.469950 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.471150 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.471632 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.471735 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.475418 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.475426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.475437 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.475467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.478179 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.478836 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.478842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.478845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.478973 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.479359 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.479365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.479368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.479686 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.480011 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.480340 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.480591 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":823,"threads_started":1}
W20260929 15:57:25.482220 5744 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57284
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:25.525657 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.510436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.5.117.1:41239)
0929 15:57:25.510662 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.510668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.510920 (+ 252us) server_negotiation.cc:407] Connection header received
0929 15:57:25.511134 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.511139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.511224 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.511328 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.513084 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.513759 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.515320 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.515711 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.515835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.520438 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.520442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.520445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.520473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.523171 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.523787 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.523794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.523797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.523931 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.524332 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.524340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.524343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.524584 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.524841 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.525117 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.525334 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:57:25.527056 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 5701
I20260929 15:57:25.538597 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.527958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.5.117.1:41239)
0929 15:57:25.528142 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.528146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.528418 (+ 272us) server_negotiation.cc:407] Connection header received
0929 15:57:25.528519 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.528522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.528568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.528640 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.529671 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.530355 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.531472 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.531892 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.531978 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.534199 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.534203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.534206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.534247 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.536715 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.537238 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.537241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.537243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.537356 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.537632 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.537635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.537637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.537864 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.538062 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.538237 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.538363 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 15:57:25.539319 5824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:25.632766 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 5701
I20260929 15:57:25.651422 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 5835
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:25.669967 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.652621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.5.117.2:34801)
0929 15:57:25.653301 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.653310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.653340 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:57:25.653427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.653435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.653696 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.653848 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.655337 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.656764 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.658050 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.658646 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.658763 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.663858 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.663871 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.663897 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.663934 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.667016 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.667721 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.667728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.667731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.667869 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.668324 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.668328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.668330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.668734 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.669000 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.669262 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.669488 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":365,"threads_started":1}
I20260929 15:57:25.671451 5958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:25.758464 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 5835
I20260929 15:57:25.776471 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 5969
Sep 29 15:57:25 dist-test-slave-b69v krb5kdc[5595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:25.793642 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:25.777934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.5.117.3:46017)
0929 15:57:25.778665 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:25.778673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:25.778707 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:57:25.778787 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:25.778794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:25.779052 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:25.779199 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:25.780547 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.781597 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.782857 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:25.783459 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:25.783556 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:25.787976 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:25.787986 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:25.788001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:25.788039 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:25.790948 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.791547 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.791554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.791557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.791686 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:25.792014 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:25.792017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:25.792019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:25.792458 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:25.792710 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:25.792961 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 15:57:25.793234 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":370,"threads_started":1}
I20260929 15:57:25.795174 6092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:25.887045 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 5969
I20260929 15:57:25.905824 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 5608
I20260929 15:57:25.907495 5670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:26.021816 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 5608
2026-09-29T15:57:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4262 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:26 dist-test-slave-b69v krb5kdc[6119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:26 dist-test-slave-b69v krb5kdc[6119](info): set up 2 sockets
Sep 29 15:57:26 dist-test-slave-b69v krb5kdc[6119](info): commencing operation
krb5kdc: starting...
W20260929 15:57:27.096645 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:27 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:27.273424 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:33695
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:33695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:27.621680 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:27.621985 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:27.622179 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:27.629359 6132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:27.629453 6132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:27.629551 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:27.629662 6132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:27.629756 6132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:27.639302 6132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:33695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:27.640697 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:27.642129 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:27.655898 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.655985 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.655901 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:27.657063 6132 server_base.cc:1034] running on GCE node
I20260929 15:57:27.657913 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:27.659395 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:27.660616 6132 hybrid_clock.cc:648] HybridClock initialized: now 1790697447660572 us; error 39 us; skew 500 ppm
I20260929 15:57:27.661020 6132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:27 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:27.672483 6132 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:27.675592 6132 webserver.cc:460] Webserver started at http://127.5.117.62:36031/ using document root <none> and password file <none>
I20260929 15:57:27.676550 6132 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:27.676688 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:27.677081 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:27.680622 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "5f1adf9e3166451d8a7d4cd769d1efe7"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-b69v"
server_key: "41ece6d66e06af95cfaad830fffe423c"
server_key_iv: "bfc0ab608d9f472b0fc6ecafb33c90ba"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.681429 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "5f1adf9e3166451d8a7d4cd769d1efe7"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-b69v"
server_key: "41ece6d66e06af95cfaad830fffe423c"
server_key_iv: "bfc0ab608d9f472b0fc6ecafb33c90ba"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.687770 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 15:57:27.692040 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:27.694110 6132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:57:27.694425 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "5f1adf9e3166451d8a7d4cd769d1efe7"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-b69v"
server_key: "41ece6d66e06af95cfaad830fffe423c"
server_key_iv: "bfc0ab608d9f472b0fc6ecafb33c90ba"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.694707 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:27.714737 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:27.720106 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:27.720438 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:27.741369 6132 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:33695
I20260929 15:57:27.741371 6198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:33695 every 8 connection(s)
I20260929 15:57:27.743172 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:27.748232 6199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:27.750046 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6132
I20260929 15:57:27.750308 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:27.751003 5588 external_mini_cluster.cc:1428] Setting key 6bc6ccfc442c85bfe580f21ad5d46816
I20260929 15:57:27.765020 6199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7: Bootstrap starting.
Sep 29 15:57:27 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:27.769152 6199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:27.770677 6199 log.cc:824] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:27.774521 6199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7: No bootstrap required, opened a new log
I20260929 15:57:27.778303 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.753319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.5.117.62:33695)
0929 15:57:27.754234 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.754252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.754305 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:57:27.756430 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.756460 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.757006 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.757440 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.759328 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.761542 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.762988 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.764279 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.764603 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.769058 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.769070 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.769100 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.769159 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.773316 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.774067 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.774074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.774083 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.774300 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.774681 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.774686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.774689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.775471 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.775773 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.776309 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.776788 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":386,"threads_started":1}
I20260929 15:57:27.784351 6199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } }
I20260929 15:57:27.784716 6199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:27.784785 6199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1adf9e3166451d8a7d4cd769d1efe7, State: Initialized, Role: FOLLOWER
I20260929 15:57:27.785370 6199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } }
I20260929 15:57:27.785620 6199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:27.785734 6199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:27.785866 6199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:27.788302 6199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } }
I20260929 15:57:27.788801 6199 leader_election.cc:302] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f1adf9e3166451d8a7d4cd769d1efe7; no voters:
I20260929 15:57:27.789350 6199 leader_election.cc:288] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:27.789497 6204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:27.790294 6204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1adf9e3166451d8a7d4cd769d1efe7, State: Running, Role: LEADER
I20260929 15:57:27.791250 6204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } }
I20260929 15:57:27.791805 6199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:27.794777 6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } } }
I20260929 15:57:27.794891 6206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f1adf9e3166451d8a7d4cd769d1efe7. Latest consensus state: current_term: 1 leader_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1adf9e3166451d8a7d4cd769d1efe7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33695 } } }
I20260929 15:57:27.795190 6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:27.795282 6206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:27.797725 6212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:27.804700 6212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:27.816967 6212 catalog_manager.cc:1347] Generated new cluster ID: c237566dbbab45cab9b9cd42d02fbd4e
I20260929 15:57:27.817162 6212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:27.840914 6212 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:27.842262 6212 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:27.847728 6212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f1adf9e3166451d8a7d4cd769d1efe7: Generated new TSK 0
I20260929 15:57:27.848594 6212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:27.903450 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:28.279943 6225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:28.280356 6225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:28.280552 6225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:28.289070 6225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:28.289242 6225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:28.289426 6225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:28.300552 6225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:28.302001 6225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:28.304025 6225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:28.319334 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:28.319310 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:28.319511 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:28.326107 6225 server_base.cc:1034] running on GCE node
I20260929 15:57:28.327050 6225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:28.328413 6225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:28.329722 6225 hybrid_clock.cc:648] HybridClock initialized: now 1790697448329670 us; error 45 us; skew 500 ppm
I20260929 15:57:28.330233 6225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:28 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697448, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:28.334432 6225 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:28.336364 6225 webserver.cc:460] Webserver started at http://127.5.117.1:45629/ using document root <none> and password file <none>
I20260929 15:57:28.337239 6225 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:28.337354 6225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:28.337666 6225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:28.341246 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "c19afd4907e941e58895e032fe8908c4"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "cb870de18a4320205e2a3a809c4da4af"
server_key_iv: "338577107763e6c91c1d86a9ae446325"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.342139 6225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "c19afd4907e941e58895e032fe8908c4"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "cb870de18a4320205e2a3a809c4da4af"
server_key_iv: "338577107763e6c91c1d86a9ae446325"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.348259 6225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:57:28.352145 6240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.353773 6225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:28.353940 6225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "c19afd4907e941e58895e032fe8908c4"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "cb870de18a4320205e2a3a809c4da4af"
server_key_iv: "338577107763e6c91c1d86a9ae446325"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.354159 6225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:28.383657 6225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:28.388612 6225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:28.388890 6225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:28.389918 6225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:28.391357 6225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:28.391434 6225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.391523 6225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:28.391569 6225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.430853 6225 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:34251
I20260929 15:57:28.430955 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:34251 every 8 connection(s)
I20260929 15:57:28.432400 6225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:28.441766 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6225
I20260929 15:57:28.442058 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:28.442770 5588 external_mini_cluster.cc:1428] Setting key e1ad27cba0690a0a740010aab6678e85
Sep 29 15:57:28 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697448, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:28.454164 6355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.435462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:37221 (local address 127.5.117.62:33695)
0929 15:57:28.436260 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.436265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.437048 (+ 783us) server_negotiation.cc:407] Connection header received
0929 15:57:28.439008 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.439014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.439117 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.439251 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:28.440881 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.441700 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.443561 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.443981 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.444204 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.448098 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:28.448111 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:28.448115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:28.448149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:28.451012 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:28.451737 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:28.451742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:28.451746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:28.451875 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:28.452362 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:28.452370 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:28.452373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:28.452650 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:28.452815 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.453552 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.453779 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":495,"threads_started":1}
I20260929 15:57:28.455194 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.436002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:33695 (local address 127.5.117.1:37221)
0929 15:57:28.436852 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.436894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.438710 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.439491 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.439499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.440042 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:28.440705 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.440723 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.441861 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.441868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.443364 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.443389 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.444294 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.444301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.444586 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.445546 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:28.445570 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:28.447827 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:28.451159 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:28.451168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:28.451183 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:28.451609 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:28.451986 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:28.452001 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:28.452004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:28.452239 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:28.452781 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:28.452789 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:28.453154 (+ 365us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.453424 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.453955 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":330,"threads_started":1}
I20260929 15:57:28.457149 6353 heartbeater.cc:347] Connected to a master server at 127.5.117.62:33695
I20260929 15:57:28.457520 6353 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:28.458576 6353 heartbeater.cc:511] Master 127.5.117.62:33695 requested a full tablet report, sending...
I20260929 15:57:28.461512 6164 ts_manager.cc:194] Registered new tserver with Master: c19afd4907e941e58895e032fe8908c4 (127.5.117.1:34251)
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 15:57:28.463303 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:37221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:28.488600 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:28.850409 6359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:28.850729 6359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:28.850915 6359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:28.858026 6359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:28.858160 6359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:28.858343 6359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:28.867695 6359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:28.869071 6359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:28.870479 6359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:28.884341 6364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:28.884307 6367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:28.884277 6365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:28.885499 6359 server_base.cc:1034] running on GCE node
I20260929 15:57:28.886080 6359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:28.886976 6359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:28.888221 6359 hybrid_clock.cc:648] HybridClock initialized: now 1790697448888167 us; error 44 us; skew 500 ppm
I20260929 15:57:28.888535 6359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:28 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697448, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:28.891641 6359 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:28.893486 6359 webserver.cc:460] Webserver started at http://127.5.117.2:39573/ using document root <none> and password file <none>
I20260929 15:57:28.894191 6359 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:28.894338 6359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:28.894675 6359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:28.897706 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "0a03dc1867ca4f5bac5d828bff34f2c5"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "3b83f84bc42e320d9631af514e0e7149"
server_key_iv: "61b05222a7ec5822ce025868b7a60c95"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.898602 6359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "0a03dc1867ca4f5bac5d828bff34f2c5"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "3b83f84bc42e320d9631af514e0e7149"
server_key_iv: "61b05222a7ec5822ce025868b7a60c95"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.903867 6359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:28.907375 6374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.908643 6359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:28.908802 6359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "0a03dc1867ca4f5bac5d828bff34f2c5"
format_stamp: "Formatted at 2026-09-29 15:57:28 on dist-test-slave-b69v"
server_key: "3b83f84bc42e320d9631af514e0e7149"
server_key_iv: "61b05222a7ec5822ce025868b7a60c95"
server_key_version: "encryptionkey@0"
I20260929 15:57:28.909014 6359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:28.938234 6359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:28.942821 6359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:28.943156 6359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:28.944288 6359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:28.945744 6359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:28.945850 6359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.946013 6359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:28.946092 6359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:28.981910 6359 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:43935
I20260929 15:57:28.981947 6486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:43935 every 8 connection(s)
I20260929 15:57:28.983403 6359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:28.990159 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6359
I20260929 15:57:28.990382 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:28.990839 5588 external_mini_cluster.cc:1428] Setting key 11a9d261ee041827bc1b857b64245b63
Sep 29 15:57:28 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697448, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:29.007120 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.986343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:59263 (local address 127.5.117.62:33695)
0929 15:57:28.986908 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.986912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.987767 (+ 855us) server_negotiation.cc:407] Connection header received
0929 15:57:28.989751 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.989757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.989853 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.989986 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:28.991750 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.992760 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.994587 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.995449 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.995608 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.999874 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:28.999881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:28.999897 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:28.999942 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.003686 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.004451 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.004460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.004466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.004636 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.005041 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.005048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.005054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.005445 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.005632 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.006319 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.006578 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":314,"threads_started":1}
I20260929 15:57:29.007820 6490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.986869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:33695 (local address 127.5.117.2:59263)
0929 15:57:28.987605 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.987644 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.989466 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.990225 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.990233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.990857 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:28.991567 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.991594 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.992955 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.992960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.994437 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.994449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.995666 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.995670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.995857 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.996729 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:28.996757 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:28.999626 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:29.003819 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:29.003826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:29.003839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:29.004332 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:29.004745 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:29.004748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:29.004750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:29.004924 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:29.005578 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:29.005585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:29.005907 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.006153 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.006544 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":310,"threads_started":1}
I20260929 15:57:29.009433 6487 heartbeater.cc:347] Connected to a master server at 127.5.117.62:33695
I20260929 15:57:29.009744 6487 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:29.010515 6487 heartbeater.cc:511] Master 127.5.117.62:33695 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:57:29.012181 6164 ts_manager.cc:194] Registered new tserver with Master: 0a03dc1867ca4f5bac5d828bff34f2c5 (127.5.117.2:43935)
I20260929 15:57:29.013190 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:59263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:29.035910 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:29.382411 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:29.382695 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:29.382865 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:29.389717 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:29.389833 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:29.390049 6493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:29.400180 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:33695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:29.401654 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:29.403178 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:29.417568 6501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:29.417682 6498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:29.417659 6499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:29.419206 6493 server_base.cc:1034] running on GCE node
I20260929 15:57:29.419960 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:29.421118 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:29.422371 6493 hybrid_clock.cc:648] HybridClock initialized: now 1790697449422313 us; error 45 us; skew 500 ppm
I20260929 15:57:29.422740 6493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:29.426291 6493 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:29.428234 6493 webserver.cc:460] Webserver started at http://127.5.117.3:44911/ using document root <none> and password file <none>
I20260929 15:57:29.429082 6493 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:29.429239 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:29.429610 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:29.433202 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "ea6876b938804e389f2ccd84a4052af4"
format_stamp: "Formatted at 2026-09-29 15:57:29 on dist-test-slave-b69v"
server_key: "2c992cbb0e2e9362b8b397240464602c"
server_key_iv: "469aad530d84c87ebb1b93d2597e1e3e"
server_key_version: "encryptionkey@0"
I20260929 15:57:29.434157 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "ea6876b938804e389f2ccd84a4052af4"
format_stamp: "Formatted at 2026-09-29 15:57:29 on dist-test-slave-b69v"
server_key: "2c992cbb0e2e9362b8b397240464602c"
server_key_iv: "469aad530d84c87ebb1b93d2597e1e3e"
server_key_version: "encryptionkey@0"
I20260929 15:57:29.440253 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:57:29.444052 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:29.445556 6493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:29.445736 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "ea6876b938804e389f2ccd84a4052af4"
format_stamp: "Formatted at 2026-09-29 15:57:29 on dist-test-slave-b69v"
server_key: "2c992cbb0e2e9362b8b397240464602c"
server_key_iv: "469aad530d84c87ebb1b93d2597e1e3e"
server_key_version: "encryptionkey@0"
I20260929 15:57:29.445966 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:29.464706 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:29.466737 6353 heartbeater.cc:503] Master 127.5.117.62:33695 was elected leader, sending a full tablet report...
I20260929 15:57:29.470118 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:29.470438 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:29.471520 6493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:29.473167 6493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:29.473277 6493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:29.473436 6493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:29.473513 6493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:29.509369 6493 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:33945
I20260929 15:57:29.509418 6620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:33945 every 8 connection(s)
I20260929 15:57:29.510941 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:29.512084 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6493
I20260929 15:57:29.512378 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:29.513031 5588 external_mini_cluster.cc:1428] Setting key 06b306912404b9489299bd0e2e4e4a06
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:29.534605 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.514432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:39631 (local address 127.5.117.62:33695)
0929 15:57:29.515013 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.515020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.516434 (+ 1414us) server_negotiation.cc:407] Connection header received
0929 15:57:29.518744 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.518749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.518849 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.518985 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.520806 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.521671 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.523437 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.523858 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.523998 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.528179 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.528186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.528191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.528236 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.531172 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.531921 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.531927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.531932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.532103 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.532596 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.532601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.532604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.532938 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.533102 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.533901 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.534175 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":265,"threads_started":1}
I20260929 15:57:29.535802 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.515218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:33695 (local address 127.5.117.3:39631)
0929 15:57:29.516271 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.516313 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.518451 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.519254 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.519264 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.519854 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:29.520618 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.520639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.521835 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.521840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.523255 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.523269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.524138 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.524146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.524399 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.525219 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:29.525244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:29.527905 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:29.531320 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:29.531329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:29.531348 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:29.531793 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:29.532247 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:29.532252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:29.532256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:29.532471 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:29.533069 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:29.533077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:29.533476 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.533780 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.534423 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":497,"threads_started":1}
I20260929 15:57:29.537304 6621 heartbeater.cc:347] Connected to a master server at 127.5.117.62:33695
I20260929 15:57:29.537662 6621 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:29.538578 6621 heartbeater.cc:511] Master 127.5.117.62:33695 requested a full tablet report, sending...
I20260929 15:57:29.540380 6164 ts_manager.cc:194] Registered new tserver with Master: ea6876b938804e389f2ccd84a4052af4 (127.5.117.3:33945)
I20260929 15:57:29.541460 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:39631
I20260929 15:57:29.542034 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:57:29.562763 6623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.549232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.5.117.62:33695)
0929 15:57:29.549494 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.549503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.550003 (+ 500us) server_negotiation.cc:407] Connection header received
0929 15:57:29.550163 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.550167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.550240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.550326 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.551999 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.552836 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.554072 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.554462 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.554551 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.557090 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.557095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.557098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.557130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.560063 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.560864 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.560868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.560870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.560998 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.561314 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.561317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.561319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.561676 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.561871 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.562162 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.562366 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:57:29.570302 6164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:29.578217 6164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:29.605473 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.594364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:34251 (local address 127.0.0.1:36770)
0929 15:57:29.595606 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.595631 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.595749 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.597007 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.597012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.597038 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:29.597486 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.597500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.599870 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.599875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.602165 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.602199 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.604359 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.604366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.604582 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.604650 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.604775 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.605008 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":923,"threads_started":1}
I20260929 15:57:29.607437 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.594919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.5.117.1:34251)
0929 15:57:29.596383 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.596392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.596422 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:57:29.596522 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.596530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.596756 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.597735 (+ 979us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:29.598008 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.599697 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.602439 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.604130 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.606514 (+ 2384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.606858 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.606937 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.607104 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":1094,"threads_started":1}
I20260929 15:57:29.607450 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.593156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.5.117.2:43935)
0929 15:57:29.594131 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.594157 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.594184 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:29.594320 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.594327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.594601 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.594837 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:29.595629 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.597441 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.603106 (+ 5665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.604707 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.604809 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.605022 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.606825 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.607044 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":636,"threads_started":1}
I20260929 15:57:29.608282 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.592887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:43935 (local address 127.0.0.1:55438)
0929 15:57:29.593752 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.593810 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.593995 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.594839 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.594847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.594880 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:29.595370 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.595388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.598877 (+ 3489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.598883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.602882 (+ 3999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.602896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.605236 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.605242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.605428 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.605479 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.607758 (+ 2279us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.607945 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":447,"threads_started":1}
I20260929 15:57:29.613302 6288 tablet_service.cc:1467] Processing CreateTablet for tablet e59f20ea19c845759ac9d31b2405a293 (DEFAULT_TABLE table=test-table [id=c247a72f15bd4363b4aa41d028d08d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:29.613302 6422 tablet_service.cc:1467] Processing CreateTablet for tablet e59f20ea19c845759ac9d31b2405a293 (DEFAULT_TABLE table=test-table [id=c247a72f15bd4363b4aa41d028d08d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:29.615140 6288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59f20ea19c845759ac9d31b2405a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:29.615223 6422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59f20ea19c845759ac9d31b2405a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:29.617336 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.596348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:33945 (local address 127.0.0.1:54772)
0929 15:57:29.605772 (+ 9424us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.605796 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.605925 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.609844 (+ 3919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.609849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.609878 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:29.610081 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.610092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.612919 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.612926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.614436 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.614458 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.616462 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.616468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.616677 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.616756 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.616852 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.617026 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9318,"thread_start_us":11886,"threads_started":1}
I20260929 15:57:29.618217 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.596235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.5.117.3:33945)
0929 15:57:29.609105 (+ 12870us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.609115 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.609155 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:57:29.609265 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.609273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.609573 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.609868 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:29.610290 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.612687 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.614611 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.616285 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.616438 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.616704 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.617622 (+ 918us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.617826 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12640,"thread_start_us":12437,"threads_started":1}
I20260929 15:57:29.623502 6556 tablet_service.cc:1467] Processing CreateTablet for tablet e59f20ea19c845759ac9d31b2405a293 (DEFAULT_TABLE table=test-table [id=c247a72f15bd4363b4aa41d028d08d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:29.625515 6556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59f20ea19c845759ac9d31b2405a293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:29.632565 6641 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Bootstrap starting.
I20260929 15:57:29.635149 6640 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Bootstrap starting.
I20260929 15:57:29.638311 6641 tablet_bootstrap.cc:654] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:29.639120 6640 tablet_bootstrap.cc:654] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:29.639634 6641 log.cc:824] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:29.640452 6642 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Bootstrap starting.
I20260929 15:57:29.640619 6640 log.cc:824] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:29.644676 6641 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: No bootstrap required, opened a new log
I20260929 15:57:29.644692 6640 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: No bootstrap required, opened a new log
I20260929 15:57:29.644742 6642 tablet_bootstrap.cc:654] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:29.644999 6640 ts_tablet_manager.cc:1397] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 15:57:29.645113 6641 ts_tablet_manager.cc:1397] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260929 15:57:29.646288 6642 log.cc:824] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:29.649840 6642 tablet_bootstrap.cc:492] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: No bootstrap required, opened a new log
I20260929 15:57:29.650154 6642 ts_tablet_manager.cc:1397] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:57:29.654367 6641 raft_consensus.cc:348] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.654906 6641 raft_consensus.cc:374] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:29.655040 6641 raft_consensus.cc:729] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a03dc1867ca4f5bac5d828bff34f2c5, State: Initialized, Role: FOLLOWER
I20260929 15:57:29.655752 6641 consensus_queue.cc:260] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.656071 6640 raft_consensus.cc:348] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.656592 6640 raft_consensus.cc:374] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:29.656719 6640 raft_consensus.cc:729] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19afd4907e941e58895e032fe8908c4, State: Initialized, Role: FOLLOWER
I20260929 15:57:29.657056 6487 heartbeater.cc:503] Master 127.5.117.62:33695 was elected leader, sending a full tablet report...
I20260929 15:57:29.657398 6640 consensus_queue.cc:260] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.657863 6641 ts_tablet_manager.cc:1428] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260929 15:57:29.661558 6640 ts_tablet_manager.cc:1428] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260929 15:57:29.663697 6642 raft_consensus.cc:348] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.664202 6642 raft_consensus.cc:374] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:29.664330 6642 raft_consensus.cc:729] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6876b938804e389f2ccd84a4052af4, State: Initialized, Role: FOLLOWER
I20260929 15:57:29.665077 6642 consensus_queue.cc:260] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.666265 6621 heartbeater.cc:503] Master 127.5.117.62:33695 was elected leader, sending a full tablet report...
I20260929 15:57:29.666878 6642 ts_tablet_manager.cc:1428] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Time spent starting tablet: real 0.017s user 0.011s sys 0.003s
W20260929 15:57:29.685158 6354 tablet.cc:2369] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:29.721907 6648 raft_consensus.cc:482] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:29.722385 6648 raft_consensus.cc:504] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.724751 6648 leader_election.cc:288] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a03dc1867ca4f5bac5d828bff34f2c5 (127.5.117.2:43935), c19afd4907e941e58895e032fe8908c4 (127.5.117.1:34251)
I20260929 15:57:29.731918 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.725222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36919 (local address 127.5.117.2:43935)
0929 15:57:29.725434 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.725439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.725478 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:57:29.725687 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.725692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.725784 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.725902 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:29.726631 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.727628 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.729745 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.730970 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.731077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.731162 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.731575 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.731688 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:29.732002 6624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.725098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:43935 (local address 127.5.117.3:36919)
0929 15:57:29.725326 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.725350 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.725507 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.726142 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.726148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.726179 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:29.726441 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.726454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.727794 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.727817 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.729575 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.729588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.731134 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.731140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.731380 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.731430 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.731540 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.731696 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 15:57:29.733296 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.725911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:54987 (local address 127.5.117.1:34251)
0929 15:57:29.726162 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.726167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.726357 (+ 190us) server_negotiation.cc:407] Connection header received
0929 15:57:29.726574 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.726579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.726670 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.726760 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:29.727337 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.728638 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.730801 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.732008 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.732442 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.732561 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.732982 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.733097 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:57:29.733443 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.725561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:34251 (local address 127.5.117.3:54987)
0929 15:57:29.726253 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:29.726269 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:29.726403 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:29.726796 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:29.726800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:29.726823 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:29.727198 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.727209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.728788 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.728792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.730631 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:29.730648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.732588 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.732592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:29.732767 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.732814 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:57:29.732943 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:57:29.733090 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":367,"threads_started":1}
I20260929 15:57:29.733428 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59f20ea19c845759ac9d31b2405a293" candidate_uuid: "ea6876b938804e389f2ccd84a4052af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" is_pre_election: true
I20260929 15:57:29.734143 6442 raft_consensus.cc:2454] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6876b938804e389f2ccd84a4052af4 in term 0.
I20260929 15:57:29.734632 6308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59f20ea19c845759ac9d31b2405a293" candidate_uuid: "ea6876b938804e389f2ccd84a4052af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19afd4907e941e58895e032fe8908c4" is_pre_election: true
I20260929 15:57:29.734861 6512 leader_election.cc:302] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a03dc1867ca4f5bac5d828bff34f2c5, ea6876b938804e389f2ccd84a4052af4; no voters:
I20260929 15:57:29.735275 6308 raft_consensus.cc:2454] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea6876b938804e389f2ccd84a4052af4 in term 0.
I20260929 15:57:29.735335 6648 raft_consensus.cc:2781] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:29.735528 6648 raft_consensus.cc:482] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:29.735654 6648 raft_consensus.cc:3037] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:57:29.735661 6488 tablet.cc:2369] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:29.738499 6648 raft_consensus.cc:504] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.739166 6648 leader_election.cc:288] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [CANDIDATE]: Term 1 election: Requested vote from peers 0a03dc1867ca4f5bac5d828bff34f2c5 (127.5.117.2:43935), c19afd4907e941e58895e032fe8908c4 (127.5.117.1:34251)
I20260929 15:57:29.739698 6442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59f20ea19c845759ac9d31b2405a293" candidate_uuid: "ea6876b938804e389f2ccd84a4052af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5"
I20260929 15:57:29.739768 6308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59f20ea19c845759ac9d31b2405a293" candidate_uuid: "ea6876b938804e389f2ccd84a4052af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19afd4907e941e58895e032fe8908c4"
I20260929 15:57:29.739971 6442 raft_consensus.cc:3037] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:29.740032 6308 raft_consensus.cc:3037] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:29.742771 6308 raft_consensus.cc:2454] T e59f20ea19c845759ac9d31b2405a293 P c19afd4907e941e58895e032fe8908c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6876b938804e389f2ccd84a4052af4 in term 1.
I20260929 15:57:29.743294 6512 leader_election.cc:302] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19afd4907e941e58895e032fe8908c4, ea6876b938804e389f2ccd84a4052af4; no voters:
I20260929 15:57:29.743569 6442 raft_consensus.cc:2454] T e59f20ea19c845759ac9d31b2405a293 P 0a03dc1867ca4f5bac5d828bff34f2c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea6876b938804e389f2ccd84a4052af4 in term 1.
I20260929 15:57:29.743660 6648 raft_consensus.cc:2781] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:29.744273 6648 raft_consensus.cc:686] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [term 1 LEADER]: Becoming Leader. State: Replica: ea6876b938804e389f2ccd84a4052af4, State: Running, Role: LEADER
I20260929 15:57:29.745005 6648 consensus_queue.cc:237] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } }
I20260929 15:57:29.750638 6164 catalog_manager.cc:5495] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea6876b938804e389f2ccd84a4052af4 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "ea6876b938804e389f2ccd84a4052af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a03dc1867ca4f5bac5d828bff34f2c5" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c19afd4907e941e58895e032fe8908c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea6876b938804e389f2ccd84a4052af4" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 33945 } health_report { overall_health: HEALTHY } } }
W20260929 15:57:29.762966 6622 tablet.cc:2369] T e59f20ea19c845759ac9d31b2405a293 P ea6876b938804e389f2ccd84a4052af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:29.812669 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.795052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.5.117.1:34251)
0929 15:57:29.795332 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.795341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.795947 (+ 606us) server_negotiation.cc:407] Connection header received
0929 15:57:29.796311 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.796316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.796412 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.796536 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.798805 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.799783 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.801289 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.801761 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.801908 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.806361 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.806383 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.806403 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.806449 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.809635 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.810325 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.810332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.810335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.810483 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.810874 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.810878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.810886 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.811256 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.811612 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.811950 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.812247 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:29.873860 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.855531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.5.117.1:34251)
0929 15:57:29.855901 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.855911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.856430 (+ 519us) server_negotiation.cc:407] Connection header received
0929 15:57:29.856731 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.856736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.856821 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.856942 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.858583 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.859454 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.860883 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.861445 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.861600 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.866808 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.866816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.866820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.866873 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.870232 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.871242 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.871248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.871251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.871453 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.872319 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.872323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.872326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.872661 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.872907 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.873261 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.873483 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:29 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:29.928838 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.913670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.5.117.1:34251)
0929 15:57:29.913993 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.914000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.914261 (+ 261us) server_negotiation.cc:407] Connection header received
0929 15:57:29.914447 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.914452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.914532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.914640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.916150 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.916892 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.918176 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.918713 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.918817 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.923130 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.923136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.923140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.923172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.926197 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.926921 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.926925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.926928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.927049 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.927377 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.927380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.927382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.927676 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.927880 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.928193 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.928368 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260929 15:57:29.930965 6268 tablet_service.cc:2908] Rejecting scan request for tablet e59f20ea19c845759ac9d31b2405a293: Uninitialized: clean time has not yet been initialized
I20260929 15:57:29.934293 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6225
I20260929 15:57:29.947750 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:29.935747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.5.117.1:34251)
0929 15:57:29.935929 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:29.935935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:29.936386 (+ 451us) server_negotiation.cc:407] Connection header received
0929 15:57:29.936482 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:29.936489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:29.936592 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:29.936703 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:29.938171 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.938844 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.939978 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:29.940469 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:29.940592 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:29.942803 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:29.942809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:29.942812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:29.942848 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:29.945365 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.946021 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.946025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.946027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.946146 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:29.946448 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:29.946451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:29.946453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:29.946743 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:29.947063 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:29.947209 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 15:57:29.947368 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:29.948552 6348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:30.051044 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6225
I20260929 15:57:30.074395 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6359
Sep 29 15:57:30 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:30.091171 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:30.075467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.5.117.2:43935)
0929 15:57:30.075705 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:30.075710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:30.075731 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:30.075844 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:30.075849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:30.075956 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:30.076126 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:30.077382 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.078252 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.079605 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.080030 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.080205 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:30.084962 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:30.084975 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:30.084993 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:30.085033 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:30.088218 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.088923 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.088932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.088934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.089073 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.089471 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.089478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.089482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.089902 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:30.090239 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:30.090565 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 15:57:30.090805 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:57:30.092022 6482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:30.189405 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6359
I20260929 15:57:30.209041 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6493
Sep 29 15:57:30 dist-test-slave-b69v krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:30.226593 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:30.209965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.5.117.3:33945)
0929 15:57:30.210546 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:30.210551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:30.210568 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:57:30.210632 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:30.210636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:30.210706 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:30.210837 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:30.212321 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.213140 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.214504 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.214926 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.215072 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:30.220405 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:30.220424 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:30.220449 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:30.220502 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:30.223741 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.224639 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.224648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.224651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.224814 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.225190 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.225194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.225196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.225591 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:30.225843 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:30.226053 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:57:30.226269 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":295,"threads_started":1}
I20260929 15:57:30.227355 6616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:30.321297 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6493
I20260929 15:57:30.340309 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6132
I20260929 15:57:30.341790 6194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:30.463150 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6132
2026-09-29T15:57:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4440 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:30 dist-test-slave-b69v krb5kdc[6670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:30 dist-test-slave-b69v krb5kdc[6670](info): set up 2 sockets
Sep 29 15:57:30 dist-test-slave-b69v krb5kdc[6670](info): commencing operation
krb5kdc: starting...
W20260929 15:57:31.536550 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:31 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:31.723461 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34233
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:34233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:32.088863 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:32.089286 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:32.089454 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:32.097219 6683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:32.097374 6683 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:57:32.097483 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:32.097597 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:32.097687 6683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:32.097802 6683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:32.108335 6683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:34233
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:32.109918 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:32.111523 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:32.125327 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:32.125416 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:32.125886 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:32.126081 6683 server_base.cc:1034] running on GCE node
I20260929 15:57:32.126956 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:32.128610 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:32.129840 6683 hybrid_clock.cc:648] HybridClock initialized: now 1790697452129787 us; error 38 us; skew 500 ppm
I20260929 15:57:32.130232 6683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:32 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697452, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:32.133690 6683 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:32.135239 6683 webserver.cc:460] Webserver started at http://127.5.117.62:38265/ using document root <none> and password file <none>
I20260929 15:57:32.135943 6683 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:32.136037 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:32.136410 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:32.139506 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "174fd01a78cb47ee8904b44e6152fe99"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "91c8f95048464379e1e498dbafe08df2"
server_key_iv: "93185c6cb446cb0c65872b8353a111a6"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.140368 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "174fd01a78cb47ee8904b44e6152fe99"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "91c8f95048464379e1e498dbafe08df2"
server_key_iv: "93185c6cb446cb0c65872b8353a111a6"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.145862 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:57:32.149418 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:32.150804 6683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:32.150998 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "174fd01a78cb47ee8904b44e6152fe99"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "91c8f95048464379e1e498dbafe08df2"
server_key_iv: "93185c6cb446cb0c65872b8353a111a6"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.151217 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:32.187326 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:32.191264 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:32.191534 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:32.213951 6683 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:34233
I20260929 15:57:32.213967 6759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:34233 every 8 connection(s)
I20260929 15:57:32.215366 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:32.219336 6760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:32.221184 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6683
I20260929 15:57:32.221427 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:32.222357 5588 external_mini_cluster.cc:1428] Setting key bbe2d37a626c6953cbceb2f185caa7d8
I20260929 15:57:32.234737 6760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Bootstrap starting.
Sep 29 15:57:32 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:32.239082 6760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:32.240638 6760 log.cc:824] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:32.243665 6760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: No bootstrap required, opened a new log
I20260929 15:57:32.248014 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.224652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.5.117.62:34233)
0929 15:57:32.225511 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.225523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.225570 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:57:32.227227 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.227247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.227629 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.227981 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.229838 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.231116 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.232897 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.233545 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.233672 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.238721 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.238734 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.238752 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.238803 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.242648 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.243316 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.243323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.243328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.243515 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.243919 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.243924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.243927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.244726 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.245058 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.245817 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.246349 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":432,"threads_started":1}
I20260929 15:57:32.253352 6760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } }
I20260929 15:57:32.253739 6760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:32.253868 6760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174fd01a78cb47ee8904b44e6152fe99, State: Initialized, Role: FOLLOWER
I20260929 15:57:32.254585 6760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } }
I20260929 15:57:32.254956 6760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:32.255112 6760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:32.255319 6760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:32.257818 6760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } }
I20260929 15:57:32.258388 6760 leader_election.cc:302] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 174fd01a78cb47ee8904b44e6152fe99; no voters:
I20260929 15:57:32.259052 6760 leader_election.cc:288] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:32.259228 6765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:32.260586 6765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [term 1 LEADER]: Becoming Leader. State: Replica: 174fd01a78cb47ee8904b44e6152fe99, State: Running, Role: LEADER
I20260929 15:57:32.261235 6765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } }
I20260929 15:57:32.261627 6760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:32.264820 6766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "174fd01a78cb47ee8904b44e6152fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } } }
I20260929 15:57:32.265796 6766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:32.265631 6767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 174fd01a78cb47ee8904b44e6152fe99. Latest consensus state: current_term: 1 leader_uuid: "174fd01a78cb47ee8904b44e6152fe99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174fd01a78cb47ee8904b44e6152fe99" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34233 } } }
I20260929 15:57:32.266830 6767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:32.267268 6774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:32.275275 6774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:32.284003 6774 catalog_manager.cc:1347] Generated new cluster ID: 1927d9a4fb1d49158c64b4a5aa5f129d
I20260929 15:57:32.284132 6774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:32.307013 6774 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:32.308451 6774 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:32.315197 6774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Generated new TSK 0
I20260929 15:57:32.316260 6774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:32.371248 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34233
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:32.741852 6786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:32.742177 6786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:32.742354 6786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:32.749720 6786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:32.749866 6786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:32.750032 6786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:32.750176 6786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:32.760661 6786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:32.762316 6786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:32.764115 6786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:32.779439 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:32.779551 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:32.779395 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:32.785831 6786 server_base.cc:1034] running on GCE node
I20260929 15:57:32.786612 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:32.787678 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:32.788964 6786 hybrid_clock.cc:648] HybridClock initialized: now 1790697452788893 us; error 52 us; skew 500 ppm
I20260929 15:57:32.789363 6786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:32 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697452, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:32.797745 6786 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:32.799876 6786 webserver.cc:460] Webserver started at http://127.5.117.1:46175/ using document root <none> and password file <none>
I20260929 15:57:32.800879 6786 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:32.800987 6786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:32.801324 6786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:32.805011 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "f2d0da34e8d442288d534be261b51db8"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "43e39baba43daa5fa5e9445f13ae8c07"
server_key_iv: "c24b8cef5749e5a36a03356030a9f423"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.806105 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "f2d0da34e8d442288d534be261b51db8"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "43e39baba43daa5fa5e9445f13ae8c07"
server_key_iv: "c24b8cef5749e5a36a03356030a9f423"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.812743 6786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:57:32.817840 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:32.819655 6786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:57:32.819846 6786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "f2d0da34e8d442288d534be261b51db8"
format_stamp: "Formatted at 2026-09-29 15:57:32 on dist-test-slave-b69v"
server_key: "43e39baba43daa5fa5e9445f13ae8c07"
server_key_iv: "c24b8cef5749e5a36a03356030a9f423"
server_key_version: "encryptionkey@0"
I20260929 15:57:32.820181 6786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:32.855441 6786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:32.860400 6786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:32.860704 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:32.864305 6786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:32.864395 6786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:32.864570 6786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:32.864646 6786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:32 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697452, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:32.890501 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.865845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:42479 (local address 127.5.117.62:34233)
0929 15:57:32.866438 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.866446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.867826 (+ 1380us) server_negotiation.cc:407] Connection header received
0929 15:57:32.870755 (+ 2929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.870762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.870879 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.871032 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.873691 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.874523 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.876283 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.876933 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.877092 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.882873 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.882881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.882889 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.882947 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.886402 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.887176 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.887187 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.887190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.887319 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.887797 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.887806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.887808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.888181 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.888352 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.889532 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.889906 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
I20260929 15:57:32.891435 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.866588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.1:42479)
0929 15:57:32.867608 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:32.867673 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:32.870348 (+ 2675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:32.871397 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:32.871408 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:32.872426 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:32.873485 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:32.873516 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.874704 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.874710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:32.876135 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:32.876147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.877220 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.877227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:32.877613 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.878622 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:32.878658 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:32.882617 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:32.886554 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:32.886564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:32.886583 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:32.887039 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:32.887436 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:32.887442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:32.887446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:32.887670 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:32.888322 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:32.888330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:32.888969 (+ 639us) client_negotiation.cc:769] Sending connection context
0929 15:57:32.889353 (+ 384us) client_negotiation.cc:240] Negotiation successful
0929 15:57:32.889904 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":435,"threads_started":1}
I20260929 15:57:32.917658 6786 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:45749
I20260929 15:57:32.917618 6921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:45749 every 8 connection(s)
I20260929 15:57:32.919643 6786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:32.920529 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.901410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.1:60009)
0929 15:57:32.902232 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:32.902265 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:32.902481 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:32.903021 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:32.903027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:32.903935 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:32.904593 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:32.904609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.906406 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.906413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:32.908436 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:32.908459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.909523 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.909537 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:32.909777 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.910752 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:32.910787 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:32.912452 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:32.917092 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:32.917098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:32.917102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:32.917848 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:32.918287 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:32.918292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:32.918296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:32.918463 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:32.919028 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:32.919034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:32.919283 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 15:57:32.919443 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:57:32.919885 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":415,"threads_started":1}
I20260929 15:57:32.920835 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.901426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:60009 (local address 127.5.117.62:34233)
0929 15:57:32.901718 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.901728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.902395 (+ 667us) server_negotiation.cc:407] Connection header received
0929 15:57:32.902701 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.902708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.902819 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.902962 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.904817 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.906183 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.908649 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.909267 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.909435 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.912737 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.912746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.912755 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.912815 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.916830 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.918012 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.918017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.918019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.918165 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.918581 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.918585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.918587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.918901 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.919023 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.919942 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.920382 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:32.922485 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6786
I20260929 15:57:32.922739 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:32.923463 5588 external_mini_cluster.cc:1428] Setting key 69c9b1818e1780758fc36e753984a62d
I20260929 15:57:32.924203 6922 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34233
I20260929 15:57:32.924484 6922 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:32.925670 6922 heartbeater.cc:511] Master 127.5.117.62:34233 requested a full tablet report, sending...
I20260929 15:57:32.928524 6715 ts_manager.cc:194] Registered new tserver with Master: f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749)
I20260929 15:57:32.930214 6715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:42479
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:32.973819 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34233
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:33.351629 6926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:33.351969 6926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:33.352174 6926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:33.360183 6926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:33.360303 6926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:33.360460 6926 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:33.360596 6926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:33.373564 6926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:33.375079 6926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:33.376581 6926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:33.391602 6932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:33.391551 6934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:33.392015 6931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:33.393800 6926 server_base.cc:1034] running on GCE node
I20260929 15:57:33.394624 6926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:33.395733 6926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:33.396997 6926 hybrid_clock.cc:648] HybridClock initialized: now 1790697453396911 us; error 60 us; skew 500 ppm
I20260929 15:57:33.397349 6926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:33 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697453, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:33.400961 6926 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:33.402580 6926 webserver.cc:460] Webserver started at http://127.5.117.2:43511/ using document root <none> and password file <none>
I20260929 15:57:33.403388 6926 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:33.403525 6926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:33.403893 6926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:33.407217 6926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "9435f094fd87477b902ec6855fd44a20"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "7417c7a5687e7e84dbe2af25f01436ee"
server_key_iv: "a4954740382b597c41f68cdfd3ecb8df"
server_key_version: "encryptionkey@0"
I20260929 15:57:33.408186 6926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "9435f094fd87477b902ec6855fd44a20"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "7417c7a5687e7e84dbe2af25f01436ee"
server_key_iv: "a4954740382b597c41f68cdfd3ecb8df"
server_key_version: "encryptionkey@0"
I20260929 15:57:33.413821 6926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:33.417676 6941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:33.419296 6926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:33.419508 6926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "9435f094fd87477b902ec6855fd44a20"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "7417c7a5687e7e84dbe2af25f01436ee"
server_key_iv: "a4954740382b597c41f68cdfd3ecb8df"
server_key_version: "encryptionkey@0"
I20260929 15:57:33.419771 6926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:33.440383 6926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:33.446502 6926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:33.446858 6926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:33.450246 6926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:33.450423 6926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:33.450623 6926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:33.450721 6926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:33 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697453, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:33.484975 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:33.451433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35503 (local address 127.5.117.62:34233)
0929 15:57:33.451976 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:33.451985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:33.453898 (+ 1913us) server_negotiation.cc:407] Connection header received
0929 15:57:33.458548 (+ 4650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:33.458560 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:33.458658 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:33.458747 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:33.461019 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.462682 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.466780 (+ 4098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.467638 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.467810 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:33.475068 (+ 7258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:33.475078 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:33.475086 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:33.475157 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:33.479608 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:33.480759 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:33.480765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:33.480769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:33.480911 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:33.481573 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:33.481578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:33.481582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:33.482081 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:33.482269 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:33.484271 (+ 2002us) server_negotiation.cc:299] Negotiation successful
0929 15:57:33.484534 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260929 15:57:33.486114 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:33.452320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.2:35503)
0929 15:57:33.453602 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:33.453677 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:33.457904 (+ 4227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:33.459413 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:33.459422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:33.460100 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:33.460812 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:33.460832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.462998 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.463007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:33.466560 (+ 3553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:33.466583 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.468023 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.468035 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:33.468486 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:33.469596 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:33.469656 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:33.474675 (+ 5019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:33.479779 (+ 5104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:33.479791 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:33.479814 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:33.480622 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:33.481102 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:33.481110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:33.481116 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:33.481435 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:33.482230 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:33.482244 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:33.482876 (+ 632us) client_negotiation.cc:769] Sending connection context
0929 15:57:33.483353 (+ 477us) client_negotiation.cc:240] Negotiation successful
0929 15:57:33.484194 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":669,"threads_started":1}
I20260929 15:57:33.500394 6926 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:36003
I20260929 15:57:33.500422 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:36003 every 8 connection(s)
I20260929 15:57:33.501976 6926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:33.504664 7062 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34233
I20260929 15:57:33.504946 7062 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:33.505896 7062 heartbeater.cc:511] Master 127.5.117.62:34233 requested a full tablet report, sending...
I20260929 15:57:33.506450 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 6926
I20260929 15:57:33.506698 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:33.507365 5588 external_mini_cluster.cc:1428] Setting key 5e3ded8f425454aef1c8850fda3e1cc4
I20260929 15:57:33.507864 6715 ts_manager.cc:194] Registered new tserver with Master: 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003)
I20260929 15:57:33.509294 6715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:35503
I20260929 15:57:33.512547 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:33.497284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:50921 (local address 127.5.117.62:34233)
0929 15:57:33.497556 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:33.497566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:33.497845 (+ 279us) server_negotiation.cc:407] Connection header received
0929 15:57:33.498115 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:33.498121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:33.498213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:33.498331 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:33.499589 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.500707 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.502075 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.502535 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.502678 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:33.505796 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:33.505801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:33.505804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:33.505837 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:33.509457 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:33.510367 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:33.510375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:33.510379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:33.510522 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:33.510919 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:33.510925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:33.510929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:33.511301 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:33.511476 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:33.511900 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 15:57:33.512179 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 15:57:33.512709 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:33.496718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.2:50921)
0929 15:57:33.497707 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:33.497731 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:33.497951 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:33.498359 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:33.498364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:33.498897 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:33.499332 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:33.499354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.500849 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.500855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:33.501910 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:33.501923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:33.502764 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:33.502773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:33.502968 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:33.504115 (+ 1147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:33.504138 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:33.505608 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:33.509600 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:33.509606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:33.509610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:33.510241 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:33.510651 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:33.510656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:33.510660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:33.510807 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:33.511429 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:33.511434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:33.511640 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 15:57:33.511792 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 15:57:33.512182 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":648,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:33.559419 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34233
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:33.932014 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:33.932421 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:33.932622 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:57:33.934067 6922 heartbeater.cc:503] Master 127.5.117.62:34233 was elected leader, sending a full tablet report...
W20260929 15:57:33.940830 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:33.940990 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:33.941145 7066 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:33.941293 7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:33.952313 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:34233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:33.954622 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:33.956616 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:33.971987 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:33.972070 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:33.971899 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:33.973145 7066 server_base.cc:1034] running on GCE node
I20260929 15:57:33.973826 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:33.974897 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:33.976197 7066 hybrid_clock.cc:648] HybridClock initialized: now 1790697453976134 us; error 99 us; skew 500 ppm
I20260929 15:57:33.976568 7066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:33 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697453, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:33.980701 7066 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:33.982745 7066 webserver.cc:460] Webserver started at http://127.5.117.3:41101/ using document root <none> and password file <none>
I20260929 15:57:33.983661 7066 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:33.983764 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:33.984263 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:33.987885 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "ec5d8e34c1ee421c828aad584d27c4a3"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "017842823a94a8c6d0e82194ed226f11"
server_key_iv: "e179f8a3bde355f859de8bc230c0d7fe"
server_key_version: "encryptionkey@0"
I20260929 15:57:33.989001 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "ec5d8e34c1ee421c828aad584d27c4a3"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "017842823a94a8c6d0e82194ed226f11"
server_key_iv: "e179f8a3bde355f859de8bc230c0d7fe"
server_key_version: "encryptionkey@0"
I20260929 15:57:33.996196 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 15:57:34.001070 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:34.002897 7066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:57:34.003190 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "ec5d8e34c1ee421c828aad584d27c4a3"
format_stamp: "Formatted at 2026-09-29 15:57:33 on dist-test-slave-b69v"
server_key: "017842823a94a8c6d0e82194ed226f11"
server_key_iv: "e179f8a3bde355f859de8bc230c0d7fe"
server_key_version: "encryptionkey@0"
I20260929 15:57:34.003490 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:34.020640 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:34.026731 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:34.027066 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:34.030371 7066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:34.030462 7066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:34.030629 7066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:34.030715 7066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:34 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697453, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:34.062338 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.032705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.3:37709)
0929 15:57:34.033953 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.034007 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.037151 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.038111 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.038120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.039049 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:34.040158 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.040185 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.042097 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.042101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.044368 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.044385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.045244 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.045250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.045471 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.046369 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:34.046392 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:34.050437 (+ 4045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:34.055820 (+ 5383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.055829 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.055885 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.056690 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.057218 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.057222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.057226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.057531 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.058291 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:34.058303 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:34.058773 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.059150 (+ 377us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.060376 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":506,"threads_started":1}
I20260929 15:57:34.063884 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.031966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:37709 (local address 127.5.117.62:34233)
0929 15:57:34.032673 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.032681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.034177 (+ 1496us) server_negotiation.cc:407] Connection header received
0929 15:57:34.037540 (+ 3363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.037546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.037679 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.037816 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:34.040479 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.041909 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.044530 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.044980 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.045137 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.050754 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:34.050760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:34.050764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:34.050798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:34.055584 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.056838 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.056861 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.056866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.057079 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.057686 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.057691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.057694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.058129 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:34.058294 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.063369 (+ 5075us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.063573 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":373,"threads_started":1}
I20260929 15:57:34.091048 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.072170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:43289 (local address 127.5.117.62:34233)
0929 15:57:34.072393 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.072402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.073334 (+ 932us) server_negotiation.cc:407] Connection header received
0929 15:57:34.073569 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.073578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.073685 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.073792 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:34.074911 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.076295 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.079141 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.079955 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.080154 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.083104 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:34.083109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:34.083113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:34.083145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:34.087399 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.088667 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.088675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.088680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.088850 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.089338 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.089345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.089348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.089756 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:34.089916 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.090400 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.090645 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:57:34.091132 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.072194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.5.117.3:43289)
0929 15:57:34.073221 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.073238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.073416 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.073980 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.073983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.074447 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:34.074763 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.074772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.076478 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.076487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.078948 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.078980 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.080227 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.080236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.080501 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.081439 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:34.081468 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:34.082875 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:34.087573 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.087582 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.087595 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.088498 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.089013 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.089020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.089024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.089201 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.089979 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:34.089982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:34.090142 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.090293 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.090682 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":521,"threads_started":1}
I20260929 15:57:34.093245 7066 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43277
I20260929 15:57:34.093302 7201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43277 every 8 connection(s)
I20260929 15:57:34.094905 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:34.097249 7202 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34233
I20260929 15:57:34.097478 7202 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:34.098169 7202 heartbeater.cc:511] Master 127.5.117.62:34233 requested a full tablet report, sending...
I20260929 15:57:34.099601 6714 ts_manager.cc:194] Registered new tserver with Master: ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3:43277)
I20260929 15:57:34.100613 6714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:37709
I20260929 15:57:34.104277 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 7066
I20260929 15:57:34.104487 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:34.104964 5588 external_mini_cluster.cc:1428] Setting key 2b5268a810be82ecfac20bbec708453b
I20260929 15:57:34.107764 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:34 dist-test-slave-b69v krb5kdc[6670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:34 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:34.159157 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.144856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.5.117.62:34233)
0929 15:57:34.145067 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.145072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.145396 (+ 324us) server_negotiation.cc:407] Connection header received
0929 15:57:34.145571 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.145576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.145671 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.145777 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:34.147327 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.148177 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.149398 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.149800 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.149893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.153937 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:34.153941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:34.153945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:34.153981 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:34.156995 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.157568 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.157572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.157574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.157689 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.158001 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.158004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.158006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.158296 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:34.158514 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.158695 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.158860 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:34.163983 6714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:34.171711 6714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:34.208263 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.185803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45749 (local address 127.0.0.1:53006)
0929 15:57:34.193173 (+ 7370us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.193236 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.193538 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.200753 (+ 7215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.200758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.200787 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.201271 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.201283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.203128 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.203134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.205039 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.205050 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.207077 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.207085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.207271 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.207337 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.207490 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.207715 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6965,"mutex_wait_us":68,"thread_start_us":6777,"threads_started":1}
I20260929 15:57:34.208685 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.186327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43277 (local address 127.0.0.1:50950)
0929 15:57:34.197645 (+ 11318us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.197672 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.197837 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.199873 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.199881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.199913 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.200315 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.200324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.202863 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.202868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.205666 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.205678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.207514 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.207520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.207751 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.207779 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.207900 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.208095 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11099,"mutex_wait_us":216,"thread_start_us":11608,"threads_started":1}
I20260929 15:57:34.208936 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.186683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.5.117.3:43277)
0929 15:57:34.198903 (+ 12220us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.198915 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.198962 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:57:34.199115 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.199128 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.199546 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.200591 (+ 1045us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.200962 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.202671 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.205837 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.207257 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.207665 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.207889 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.207983 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.208198 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11964,"thread_start_us":12776,"threads_started":1}
I20260929 15:57:34.209286 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.198502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36003 (local address 127.0.0.1:51660)
0929 15:57:34.199547 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.199566 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.199688 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.200428 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.200433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.200460 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.200923 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.200950 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.202766 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.202772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.205664 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.205679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.207444 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.207452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.207690 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.207776 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.207874 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.208095 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":131,"thread_start_us":787,"threads_started":1}
I20260929 15:57:34.209371 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.186111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.5.117.1:45749)
0929 15:57:34.200159 (+ 14048us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.200169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.200200 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:57:34.200312 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.200321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.200532 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.200763 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.201481 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.202978 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.205198 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.206545 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.206681 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.206941 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.208840 (+ 1899us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.209053 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13837,"thread_start_us":13679,"threads_started":1}
I20260929 15:57:34.209524 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.194310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.5.117.2:36003)
0929 15:57:34.198523 (+ 4213us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.198534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.199827 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 15:57:34.199948 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.199956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.200210 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.200418 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.201143 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.202630 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.205874 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.207216 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.207370 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.207589 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.208360 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.208693 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3988,"thread_start_us":3808,"threads_started":1}
I20260929 15:57:34.213606 6852 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c (DEFAULT_TABLE table=test-table [id=0b3f8bcffb444b42915192b89f431bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.213615 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c (DEFAULT_TABLE table=test-table [id=0b3f8bcffb444b42915192b89f431bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.215306 6852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86aa99ebe4ef8b40b3d964da6939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.215329 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86aa99ebe4ef8b40b3d964da6939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.214864 7132 tablet_service.cc:1467] Processing CreateTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c (DEFAULT_TABLE table=test-table [id=0b3f8bcffb444b42915192b89f431bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.216403 7132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be86aa99ebe4ef8b40b3d964da6939c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.240293 7221 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Bootstrap starting.
I20260929 15:57:34.241257 7222 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Bootstrap starting.
I20260929 15:57:34.242111 7223 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Bootstrap starting.
I20260929 15:57:34.245301 7223 tablet_bootstrap.cc:654] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.245476 7222 tablet_bootstrap.cc:654] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.245467 7221 tablet_bootstrap.cc:654] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.246759 7221 log.cc:824] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:34.246759 7222 log.cc:824] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:34.247064 7223 log.cc:824] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:34.250419 7222 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: No bootstrap required, opened a new log
I20260929 15:57:34.250471 7223 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: No bootstrap required, opened a new log
I20260929 15:57:34.250733 7223 ts_tablet_manager.cc:1397] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:57:34.250746 7222 ts_tablet_manager.cc:1397] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 15:57:34.251399 7221 tablet_bootstrap.cc:492] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: No bootstrap required, opened a new log
I20260929 15:57:34.251654 7221 ts_tablet_manager.cc:1397] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 15:57:34.258613 7222 raft_consensus.cc:348] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.259114 7222 raft_consensus.cc:374] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.259225 7222 raft_consensus.cc:729] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d0da34e8d442288d534be261b51db8, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.259915 7222 consensus_queue.cc:260] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.261868 7223 raft_consensus.cc:348] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.262279 7223 raft_consensus.cc:374] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.262375 7223 raft_consensus.cc:729] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5d8e34c1ee421c828aad584d27c4a3, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.262621 7222 ts_tablet_manager.cc:1428] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 15:57:34.262745 7221 raft_consensus.cc:348] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.263077 7223 consensus_queue.cc:260] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.263273 7221 raft_consensus.cc:374] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.263370 7221 raft_consensus.cc:729] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9435f094fd87477b902ec6855fd44a20, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.264042 7221 consensus_queue.cc:260] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.264403 7202 heartbeater.cc:503] Master 127.5.117.62:34233 was elected leader, sending a full tablet report...
I20260929 15:57:34.265853 7062 heartbeater.cc:503] Master 127.5.117.62:34233 was elected leader, sending a full tablet report...
I20260929 15:57:34.265915 7221 ts_tablet_manager.cc:1428] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 15:57:34.266830 7223 ts_tablet_manager.cc:1428] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260929 15:57:34.289901 7228 raft_consensus.cc:482] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:34.290169 7228 raft_consensus.cc:504] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.291898 7228 leader_election.cc:288] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749), 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003)
I20260929 15:57:34.298767 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.292331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:46859 (local address 127.5.117.1:45749)
0929 15:57:34.292536 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.292541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.292605 (+ 64us) server_negotiation.cc:407] Connection header received
0929 15:57:34.292868 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.292876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.293013 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.293117 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.293682 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.294737 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.296795 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.297985 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.298084 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.298165 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.298496 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.298583 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:57:34.298856 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.292291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45749 (local address 127.5.117.3:46859)
0929 15:57:34.292510 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.292532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.292668 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.293298 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.293302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.293326 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.293534 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.293543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.294888 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.294893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.296627 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.296638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.298128 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.298133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.298344 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.298378 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.298471 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.298600 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:57:34.299924 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.292742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36003 (local address 127.5.117.3:60095)
0929 15:57:34.293509 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.293523 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.293669 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.294072 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.294076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.294097 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.294564 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.294577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.295949 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.295952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.297736 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.297752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.299273 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.299278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.299381 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.299402 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.299510 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.299621 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":455,"threads_started":1}
I20260929 15:57:34.299940 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.292992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:60095 (local address 127.5.117.2:36003)
0929 15:57:34.293194 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.293199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.293618 (+ 419us) server_negotiation.cc:407] Connection header received
0929 15:57:34.293797 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.293802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.293871 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.294010 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.294721 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.295803 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.297896 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.299012 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.299153 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.299275 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.299551 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.299684 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:34.299849 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86aa99ebe4ef8b40b3d964da6939c" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d0da34e8d442288d534be261b51db8" is_pre_election: true
I20260929 15:57:34.300431 6876 raft_consensus.cc:2454] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 0.
I20260929 15:57:34.301254 7082 leader_election.cc:302] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec5d8e34c1ee421c828aad584d27c4a3, f2d0da34e8d442288d534be261b51db8; no voters:
I20260929 15:57:34.301254 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86aa99ebe4ef8b40b3d964da6939c" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20" is_pre_election: true
I20260929 15:57:34.301661 7228 raft_consensus.cc:2781] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:34.301777 7228 raft_consensus.cc:482] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:34.301788 7012 raft_consensus.cc:2454] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 0.
I20260929 15:57:34.301893 7228 raft_consensus.cc:3037] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.304571 7228 raft_consensus.cc:504] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.305184 7228 leader_election.cc:288] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 election: Requested vote from peers f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749), 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003)
I20260929 15:57:34.305581 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86aa99ebe4ef8b40b3d964da6939c" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d0da34e8d442288d534be261b51db8"
I20260929 15:57:34.305732 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be86aa99ebe4ef8b40b3d964da6939c" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20"
I20260929 15:57:34.305779 6876 raft_consensus.cc:3037] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.305945 7012 raft_consensus.cc:3037] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.308256 6876 raft_consensus.cc:2454] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 1.
I20260929 15:57:34.308555 7012 raft_consensus.cc:2454] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 1.
I20260929 15:57:34.308645 7082 leader_election.cc:302] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec5d8e34c1ee421c828aad584d27c4a3, f2d0da34e8d442288d534be261b51db8; no voters:
I20260929 15:57:34.308921 7228 raft_consensus.cc:2781] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:34.309381 7228 raft_consensus.cc:686] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 LEADER]: Becoming Leader. State: Replica: ec5d8e34c1ee421c828aad584d27c4a3, State: Running, Role: LEADER
I20260929 15:57:34.309916 7228 consensus_queue.cc:237] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.314842 6714 catalog_manager.cc:5495] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } health_report { overall_health: HEALTHY } } }
I20260929 15:57:34.339161 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.333096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.5.117.3:43277)
0929 15:57:34.333332 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.333339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.333362 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:34.333721 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.333725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.333815 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.333978 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:34.334401 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.335179 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.336675 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.337128 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.337273 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.337808 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.338466 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.338644 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.338749 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.338846 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 15:57:34.347250 7203 tablet.cc:2369] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:34.349763 7012 raft_consensus.cc:1261] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Refusing update from remote peer ec5d8e34c1ee421c828aad584d27c4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.349764 6876 raft_consensus.cc:1261] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Refusing update from remote peer ec5d8e34c1ee421c828aad584d27c4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.350847 7231 consensus_queue.cc:1035] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:34.351313 7228 consensus_queue.cc:1035] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:34.390178 6715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:34.390535 6715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:34.395411 6715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Sending DeleteTablet for 3 replicas of tablet 4be86aa99ebe4ef8b40b3d964da6939c
I20260929 15:57:34.397472 7132 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:50950
I20260929 15:57:34.397533 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:51660
I20260929 15:57:34.397666 6852 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be86aa99ebe4ef8b40b3d964da6939c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:53006
I20260929 15:57:34.399580 6715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:34.400628 6715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:34.402840 7245 tablet_replica.cc:330] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: stopping tablet replica
I20260929 15:57:34.405573 7244 tablet_replica.cc:330] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: stopping tablet replica
I20260929 15:57:34.406057 7244 raft_consensus.cc:2229] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:34.406123 7246 tablet_replica.cc:330] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: stopping tablet replica
I20260929 15:57:34.406692 7246 raft_consensus.cc:2229] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:34.406736 7244 raft_consensus.cc:2258] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:34.407331 7245 raft_consensus.cc:2229] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:34.407617 7246 raft_consensus.cc:2258] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:34.408248 7245 raft_consensus.cc:2258] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:34.410539 7246 ts_tablet_manager.cc:1905] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:34.412503 7244 ts_tablet_manager.cc:1905] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:34.412597 7132 tablet_service.cc:1467] Processing CreateTablet for tablet aa223136e22145b9b802ab9c22e563ce (DEFAULT_TABLE table=test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.413203 7132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa223136e22145b9b802ab9c22e563ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.413563 6992 tablet_service.cc:1467] Processing CreateTablet for tablet aa223136e22145b9b802ab9c22e563ce (DEFAULT_TABLE table=test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.414031 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa223136e22145b9b802ab9c22e563ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.420394 7244 ts_tablet_manager.cc:1918] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:34.420598 7244 log.cc:1196] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/4be86aa99ebe4ef8b40b3d964da6939c
I20260929 15:57:34.420970 6852 tablet_service.cc:1467] Processing CreateTablet for tablet aa223136e22145b9b802ab9c22e563ce (DEFAULT_TABLE table=test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:34.421268 7244 ts_tablet_manager.cc:1939] T 4be86aa99ebe4ef8b40b3d964da6939c P 9435f094fd87477b902ec6855fd44a20: Deleting consensus metadata
I20260929 15:57:34.421715 6852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa223136e22145b9b802ab9c22e563ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.422287 7245 ts_tablet_manager.cc:1905] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:34.423548 6699 catalog_manager.cc:4841] TS 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003): tablet 4be86aa99ebe4ef8b40b3d964da6939c (table test-table [id=0b3f8bcffb444b42915192b89f431bb7]) successfully deleted
I20260929 15:57:34.424285 7223 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Bootstrap starting.
I20260929 15:57:34.424882 7221 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Bootstrap starting.
I20260929 15:57:34.426389 7223 tablet_bootstrap.cc:654] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.426918 7221 tablet_bootstrap.cc:654] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.430353 7245 ts_tablet_manager.cc:1918] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:34.430557 7245 log.cc:1196] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/4be86aa99ebe4ef8b40b3d964da6939c
I20260929 15:57:34.431201 7245 ts_tablet_manager.cc:1939] T 4be86aa99ebe4ef8b40b3d964da6939c P ec5d8e34c1ee421c828aad584d27c4a3: Deleting consensus metadata
I20260929 15:57:34.434460 6702 catalog_manager.cc:4841] TS ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3:43277): tablet 4be86aa99ebe4ef8b40b3d964da6939c (table test-table [id=0b3f8bcffb444b42915192b89f431bb7]) successfully deleted
I20260929 15:57:34.435518 7221 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: No bootstrap required, opened a new log
I20260929 15:57:34.435698 7221 ts_tablet_manager.cc:1397] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 15:57:34.436762 7221 raft_consensus.cc:348] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.437042 7221 raft_consensus.cc:374] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.437166 7221 raft_consensus.cc:729] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9435f094fd87477b902ec6855fd44a20, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.438262 7246 ts_tablet_manager.cc:1918] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:34.438758 7222 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Bootstrap starting.
I20260929 15:57:34.438766 7221 consensus_queue.cc:260] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.439386 7221 ts_tablet_manager.cc:1428] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 15:57:34.440716 7246 log.cc:1196] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/4be86aa99ebe4ef8b40b3d964da6939c
I20260929 15:57:34.440899 7222 tablet_bootstrap.cc:654] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.441484 7246 ts_tablet_manager.cc:1939] T 4be86aa99ebe4ef8b40b3d964da6939c P f2d0da34e8d442288d534be261b51db8: Deleting consensus metadata
I20260929 15:57:34.445394 6699 catalog_manager.cc:4841] TS f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749): tablet 4be86aa99ebe4ef8b40b3d964da6939c (table test-table [id=0b3f8bcffb444b42915192b89f431bb7]) successfully deleted
I20260929 15:57:34.447077 7223 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: No bootstrap required, opened a new log
I20260929 15:57:34.447329 7223 ts_tablet_manager.cc:1397] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.007s
I20260929 15:57:34.447458 7222 tablet_bootstrap.cc:492] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: No bootstrap required, opened a new log
I20260929 15:57:34.447650 7222 ts_tablet_manager.cc:1397] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 15:57:34.448652 7222 raft_consensus.cc:348] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.448678 7223 raft_consensus.cc:348] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.448979 7223 raft_consensus.cc:374] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.449142 7223 raft_consensus.cc:729] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5d8e34c1ee421c828aad584d27c4a3, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.449348 7222 raft_consensus.cc:374] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.449488 7222 raft_consensus.cc:729] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d0da34e8d442288d534be261b51db8, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.449568 7223 consensus_queue.cc:260] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.449862 7222 consensus_queue.cc:260] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.450350 7223 ts_tablet_manager.cc:1428] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:34.452214 7222 ts_tablet_manager.cc:1428] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:57:34.501950 7239 raft_consensus.cc:482] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:34.502274 7239 raft_consensus.cc:504] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.503332 7239 leader_election.cc:288] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749), 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003)
I20260929 15:57:34.504169 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa223136e22145b9b802ab9c22e563ce" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d0da34e8d442288d534be261b51db8" is_pre_election: true
I20260929 15:57:34.504221 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa223136e22145b9b802ab9c22e563ce" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20" is_pre_election: true
I20260929 15:57:34.504660 6876 raft_consensus.cc:2454] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 0.
I20260929 15:57:34.504688 7012 raft_consensus.cc:2454] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 0.
I20260929 15:57:34.505353 7082 leader_election.cc:302] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9435f094fd87477b902ec6855fd44a20, ec5d8e34c1ee421c828aad584d27c4a3; no voters:
W20260929 15:57:34.506173 7063 tablet.cc:2369] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:34.506330 7239 raft_consensus.cc:2781] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:34.506470 7239 raft_consensus.cc:482] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:34.506567 7239 raft_consensus.cc:3037] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.510378 7239 raft_consensus.cc:504] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.511343 7239 leader_election.cc:288] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 election: Requested vote from peers f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749), 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003)
I20260929 15:57:34.512247 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa223136e22145b9b802ab9c22e563ce" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20"
I20260929 15:57:34.512248 6876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa223136e22145b9b802ab9c22e563ce" candidate_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d0da34e8d442288d534be261b51db8"
I20260929 15:57:34.512562 7012 raft_consensus.cc:3037] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.512562 6876 raft_consensus.cc:3037] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.516723 6876 raft_consensus.cc:2454] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 1.
I20260929 15:57:34.516726 7012 raft_consensus.cc:2454] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5d8e34c1ee421c828aad584d27c4a3 in term 1.
I20260929 15:57:34.517547 7082 leader_election.cc:302] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec5d8e34c1ee421c828aad584d27c4a3, f2d0da34e8d442288d534be261b51db8; no voters:
I20260929 15:57:34.518010 7239 raft_consensus.cc:2781] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:34.518216 7239 raft_consensus.cc:686] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 LEADER]: Becoming Leader. State: Replica: ec5d8e34c1ee421c828aad584d27c4a3, State: Running, Role: LEADER
I20260929 15:57:34.518608 7239 consensus_queue.cc:237] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.522389 6713 catalog_manager.cc:5495] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } health_report { overall_health: HEALTHY } } }
Sep 29 15:57:34 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697452, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:34.604894 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.587454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.5.117.62:34233)
0929 15:57:34.587713 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.587725 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.588474 (+ 749us) server_negotiation.cc:407] Connection header received
0929 15:57:34.588725 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.588729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.588791 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.588896 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:34.590302 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.591121 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.592521 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.593028 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.593177 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.598051 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:34.598059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:34.598064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:34.598116 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:34.601328 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.602562 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.602567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.602569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.602725 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.603184 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.603188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.603190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.603530 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:34.603675 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.604392 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.604615 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:34.605515 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.587440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34233 (local address 127.0.0.1:54744)
0929 15:57:34.588302 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.588324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.588571 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.588919 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.588923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.589855 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:34.590154 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.590163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.591255 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.591264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.592364 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.592376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.593401 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.593409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.593560 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.594416 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:34.594434 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:34.597614 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:34.601534 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.601560 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.601572 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.602333 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.602847 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:34.602853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:34.602857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:34.603049 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:34.603694 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:34.603705 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:34.603940 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.604295 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.604815 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":132,"thread_start_us":529,"threads_started":1}
I20260929 15:57:34.613381 6713 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:57:34.615379 6713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:34.624637 6992 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfc20206d14b93be621cc1c5f777d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34236eebb164bf394c04832ff6ac63b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:34.624647 6852 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfc20206d14b93be621cc1c5f777d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34236eebb164bf394c04832ff6ac63b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:34.624869 7132 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecfc20206d14b93be621cc1c5f777d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b34236eebb164bf394c04832ff6ac63b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:34.625463 6852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfc20206d14b93be621cc1c5f777d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.625469 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfc20206d14b93be621cc1c5f777d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.625519 7132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecfc20206d14b93be621cc1c5f777d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.634902 7221 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: Bootstrap starting.
I20260929 15:57:34.634866 7223 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: Bootstrap starting.
I20260929 15:57:34.637202 7221 tablet_bootstrap.cc:654] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.637209 7223 tablet_bootstrap.cc:654] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.640421 7222 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: Bootstrap starting.
I20260929 15:57:34.640925 7221 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: No bootstrap required, opened a new log
I20260929 15:57:34.641162 7221 ts_tablet_manager.cc:1397] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:34.642161 7221 raft_consensus.cc:348] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.642455 7221 raft_consensus.cc:374] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.642549 7221 raft_consensus.cc:729] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9435f094fd87477b902ec6855fd44a20, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.642864 7222 tablet_bootstrap.cc:654] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.643004 7223 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: No bootstrap required, opened a new log
I20260929 15:57:34.642963 7221 consensus_queue.cc:260] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.643213 7223 ts_tablet_manager.cc:1397] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 15:57:34.643641 7221 ts_tablet_manager.cc:1428] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:34.643570 7229 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.644023 7229 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.644274 7223 raft_consensus.cc:348] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.644569 7223 raft_consensus.cc:374] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.644668 7223 raft_consensus.cc:729] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5d8e34c1ee421c828aad584d27c4a3, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.644958 7223 consensus_queue.cc:260] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.645754 7223 ts_tablet_manager.cc:1428] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 15:57:34.645756 7239 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.648761 7239 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.649928 7222 tablet_bootstrap.cc:492] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: No bootstrap required, opened a new log
I20260929 15:57:34.650152 7222 ts_tablet_manager.cc:1397] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260929 15:57:34.651054 7222 raft_consensus.cc:348] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.651371 7222 raft_consensus.cc:374] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.651458 7222 raft_consensus.cc:729] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d0da34e8d442288d534be261b51db8, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.651809 7222 consensus_queue.cc:260] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.652601 7222 ts_tablet_manager.cc:1428] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:57:34.652498 7227 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.652981 7227 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 15:57:34.674486 6923 tablet.cc:2369] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:34.730437 7227 raft_consensus.cc:482] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:34.730676 7227 raft_consensus.cc:504] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.732800 7227 leader_election.cc:288] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003), ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3:43277)
I20260929 15:57:34.742394 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.733356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:34707 (local address 127.5.117.3:43277)
0929 15:57:34.733634 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.733646 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.734343 (+ 697us) server_negotiation.cc:407] Connection header received
0929 15:57:34.734695 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.734701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.734785 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.734843 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.736018 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.736909 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.739980 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.741143 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.741302 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.741451 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.741837 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.742002 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:57:34.742511 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.733143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43277 (local address 127.5.117.1:34707)
0929 15:57:34.734210 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.734237 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.734499 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.735101 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.735104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.735147 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.735806 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.735819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.737072 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.737080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.739769 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.739785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.741421 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.741427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.741655 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.741693 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.741811 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.742068 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":199,"spinlock_wait_cycles":2304,"thread_start_us":707,"threads_started":1}
I20260929 15:57:34.743120 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.733767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:58135 (local address 127.5.117.2:36003)
0929 15:57:34.734010 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.734017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.734573 (+ 556us) server_negotiation.cc:407] Connection header received
0929 15:57:34.734704 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.734711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.734820 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.734929 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:34.735917 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.737188 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.739949 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.741304 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.741443 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.741556 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.742175 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.742335 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:57:34.743552 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecfc20206d14b93be621cc1c5f777d5" candidate_uuid: "f2d0da34e8d442288d534be261b51db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" is_pre_election: true
I20260929 15:57:34.743983 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.733187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36003 (local address 127.5.117.1:58135)
0929 15:57:34.734445 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.734462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.734582 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.734971 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.734976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.735001 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:34.735675 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.735691 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.737340 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.737346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.739769 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.739785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.741737 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.741744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.741961 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.742006 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.743431 (+ 1425us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.743585 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":111,"thread_start_us":982,"threads_started":1}
I20260929 15:57:34.744095 7157 raft_consensus.cc:2454] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d0da34e8d442288d534be261b51db8 in term 0.
I20260929 15:57:34.744726 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecfc20206d14b93be621cc1c5f777d5" candidate_uuid: "f2d0da34e8d442288d534be261b51db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20" is_pre_election: true
I20260929 15:57:34.744803 6805 leader_election.cc:302] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec5d8e34c1ee421c828aad584d27c4a3, f2d0da34e8d442288d534be261b51db8; no voters:
I20260929 15:57:34.745040 7012 raft_consensus.cc:2454] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d0da34e8d442288d534be261b51db8 in term 0.
I20260929 15:57:34.745316 7227 raft_consensus.cc:2781] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:34.745456 7227 raft_consensus.cc:482] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:34.745575 7227 raft_consensus.cc:3037] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.748410 7227 raft_consensus.cc:504] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.749208 7227 leader_election.cc:288] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [CANDIDATE]: Term 1 election: Requested vote from peers 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003), ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3:43277)
I20260929 15:57:34.749727 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecfc20206d14b93be621cc1c5f777d5" candidate_uuid: "f2d0da34e8d442288d534be261b51db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9435f094fd87477b902ec6855fd44a20"
I20260929 15:57:34.749758 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecfc20206d14b93be621cc1c5f777d5" candidate_uuid: "f2d0da34e8d442288d534be261b51db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5d8e34c1ee421c828aad584d27c4a3"
I20260929 15:57:34.750003 7012 raft_consensus.cc:3037] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.749991 7157 raft_consensus.cc:3037] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.753409 7012 raft_consensus.cc:2454] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d0da34e8d442288d534be261b51db8 in term 1.
I20260929 15:57:34.753758 7157 raft_consensus.cc:2454] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d0da34e8d442288d534be261b51db8 in term 1.
I20260929 15:57:34.753925 6802 leader_election.cc:302] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9435f094fd87477b902ec6855fd44a20, f2d0da34e8d442288d534be261b51db8; no voters:
I20260929 15:57:34.754401 7227 raft_consensus.cc:2781] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:34.755020 7227 raft_consensus.cc:686] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [term 1 LEADER]: Becoming Leader. State: Replica: f2d0da34e8d442288d534be261b51db8, State: Running, Role: LEADER
I20260929 15:57:34.755587 7227 consensus_queue.cc:237] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } }
I20260929 15:57:34.758411 7262 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: TxnStatusTablet state changed. Reason: New leader f2d0da34e8d442288d534be261b51db8. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.758908 7262 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:34.759891 7264 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:57:34.760196 7264 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:57:34.761968 6714 catalog_manager.cc:5495] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2d0da34e8d442288d534be261b51db8 (127.5.117.1). New cstate: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:34.793071 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.783867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.5.117.1:45749)
0929 15:57:34.784124 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.784132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.784157 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:57:34.784324 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.784330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.784641 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.785052 (+ 411us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:34.785581 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.787078 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.789307 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.790062 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.790235 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.790935 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.792130 (+ 1195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.792321 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.792488 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.792609 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:57:34.793049 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.783562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45749 (local address 127.0.0.1:53008)
0929 15:57:34.783814 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.783838 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.784037 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.785078 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.785084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.785103 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:34.785363 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.785376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.787275 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.787281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.789098 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.789115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.790292 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.790299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.790541 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.790652 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.792288 (+ 1636us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.792317 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.792448 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.792679 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:57:34.799705 7012 raft_consensus.cc:1261] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Refusing update from remote peer f2d0da34e8d442288d534be261b51db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.799841 7157 raft_consensus.cc:1261] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Refusing update from remote peer f2d0da34e8d442288d534be261b51db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.800947 7227 consensus_queue.cc:1035] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:34.801421 7262 consensus_queue.cc:1035] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:34.813807 7229 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: TxnStatusTablet state changed. Reason: New leader f2d0da34e8d442288d534be261b51db8. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.814199 7229 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.815889 7239 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: TxnStatusTablet state changed. Reason: New leader f2d0da34e8d442288d534be261b51db8. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.816362 7239 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.819408 7227 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.819842 7227 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:34.820120 7235 mvcc.cc:205] Tried to move back new op lower bound from 7334696774849052672 to 7334696774684856320. Current Snapshot: MvccSnapshot[applied={T|T < 7334696774849052672 or (T in {7334696774849052672})}]
I20260929 15:57:34.823403 7229 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.823740 7229 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P 9435f094fd87477b902ec6855fd44a20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.829778 7262 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.830248 7262 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P f2d0da34e8d442288d534be261b51db8: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:34.839206 7239 tablet_replica.cc:439] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2d0da34e8d442288d534be261b51db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 } } peers { permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 } } peers { permanent_uuid: "ec5d8e34c1ee421c828aad584d27c4a3" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43277 } } }
I20260929 15:57:34.839944 7239 tablet_replica.cc:442] T 2ecfc20206d14b93be621cc1c5f777d5 P ec5d8e34c1ee421c828aad584d27c4a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:34.854636 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36987 (local address 127.5.117.1:45749)
0929 15:57:34.847754 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.847763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.848016 (+ 253us) server_negotiation.cc:407] Connection header received
0929 15:57:34.848419 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.848425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.848542 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.848602 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:34.849806 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.850637 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.852389 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.852915 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.853089 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.853690 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.853832 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.853944 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.854119 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.854217 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:34.855185 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.847045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45749 (local address 127.5.117.3:36987)
0929 15:57:34.847870 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.847892 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.848179 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.848868 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.848872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.848886 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:34.849360 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.849372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.850794 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.850797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.852242 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.852255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.853237 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.853244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.853408 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.853482 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.853946 (+ 464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.853961 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.854092 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.854260 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":726,"threads_started":1}
I20260929 15:57:34.870472 7012 raft_consensus.cc:1261] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Refusing update from remote peer ec5d8e34c1ee421c828aad584d27c4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.870662 6876 raft_consensus.cc:1261] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Refusing update from remote peer ec5d8e34c1ee421c828aad584d27c4a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:34.871716 7228 consensus_queue.cc:1035] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9435f094fd87477b902ec6855fd44a20" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:34.873377 7239 consensus_queue.cc:1035] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d0da34e8d442288d534be261b51db8" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 15:57:34.881078 7234 mvcc.cc:205] Tried to move back new op lower bound from 7334696775139917824 to 7334696773711708160. Current Snapshot: MvccSnapshot[applied={T|T < 7334696775139917824}]
I20260929 15:57:34.934643 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.926925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45151 (local address 127.5.117.3:43277)
0929 15:57:34.927122 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.927130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.927381 (+ 251us) server_negotiation.cc:407] Connection header received
0929 15:57:34.927639 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.927646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.927736 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.927830 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:34.928916 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.930121 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.932253 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.932999 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.933134 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.933748 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.933919 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.934018 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.934171 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.934288 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:57:34.934762 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.926543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43277 (local address 127.5.117.1:45151)
0929 15:57:34.927283 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:34.927306 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:34.927477 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:34.927850 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:34.927855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:34.927874 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:34.928730 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.928746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.930255 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.930263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.932118 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:34.932135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.933189 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.933195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:34.933453 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.933536 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:34.934025 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:34.934042 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 15:57:34.934128 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 15:57:34.934298 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":586,"threads_started":1}
I20260929 15:57:34.981544 6714 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:34.981884 6714 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:34.987385 6714 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 174fd01a78cb47ee8904b44e6152fe99: Sending DeleteTablet for 3 replicas of tablet aa223136e22145b9b802ab9c22e563ce
I20260929 15:57:34.989078 6852 tablet_service.cc:1514] Processing DeleteTablet for tablet aa223136e22145b9b802ab9c22e563ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:53006
I20260929 15:57:34.989113 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet aa223136e22145b9b802ab9c22e563ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:51660
I20260929 15:57:34.992106 7132 tablet_service.cc:1514] Processing DeleteTablet for tablet aa223136e22145b9b802ab9c22e563ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:34 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:50950
I20260929 15:57:35.004777 7281 tablet_replica.cc:330] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: stopping tablet replica
I20260929 15:57:35.005074 7281 raft_consensus.cc:2229] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:35.005579 7281 raft_consensus.cc:2258] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:35.006699 7279 tablet_replica.cc:330] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: stopping tablet replica
I20260929 15:57:35.006959 7279 raft_consensus.cc:2229] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:35.007278 7279 raft_consensus.cc:2258] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:35.010097 7281 ts_tablet_manager.cc:1905] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:35.013195 7279 ts_tablet_manager.cc:1905] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:35.018267 7278 tablet_replica.cc:330] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: stopping tablet replica
I20260929 15:57:35.018543 7278 raft_consensus.cc:2229] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:35.018883 7278 raft_consensus.cc:2258] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:35.025926 7278 ts_tablet_manager.cc:1905] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:35.040969 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.997207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.5.117.1:45749)
0929 15:57:34.997404 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.997412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.003605 (+ 6193us) server_negotiation.cc:407] Connection header received
0929 15:57:35.007589 (+ 3984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.007596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.007700 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.007777 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.010264 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.011413 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.024210 (+ 12797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.024959 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.025096 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.031297 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.031336 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.031385 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.031439 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.036108 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.037065 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.037075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.037079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.037279 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.037718 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.037724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.037728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.038281 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.038806 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.040155 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.040515 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260929 15:57:35.042394 6917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53010
I20260929 15:57:35.048702 7278 ts_tablet_manager.cc:1918] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:35.048930 7278 log.cc:1196] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/aa223136e22145b9b802ab9c22e563ce
I20260929 15:57:35.049598 7278 ts_tablet_manager.cc:1939] T aa223136e22145b9b802ab9c22e563ce P 9435f094fd87477b902ec6855fd44a20: Deleting consensus metadata
I20260929 15:57:35.051175 6699 catalog_manager.cc:4841] TS 9435f094fd87477b902ec6855fd44a20 (127.5.117.2:36003): tablet aa223136e22145b9b802ab9c22e563ce (table test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]) successfully deleted
I20260929 15:57:35.053898 7281 ts_tablet_manager.cc:1918] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:35.054105 7281 log.cc:1196] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/aa223136e22145b9b802ab9c22e563ce
I20260929 15:57:35.054757 7281 ts_tablet_manager.cc:1939] T aa223136e22145b9b802ab9c22e563ce P ec5d8e34c1ee421c828aad584d27c4a3: Deleting consensus metadata
I20260929 15:57:35.056568 6702 catalog_manager.cc:4841] TS ec5d8e34c1ee421c828aad584d27c4a3 (127.5.117.3:43277): tablet aa223136e22145b9b802ab9c22e563ce (table test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]) successfully deleted
I20260929 15:57:35.058152 7279 ts_tablet_manager.cc:1918] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:35.058346 7279 log.cc:1196] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/aa223136e22145b9b802ab9c22e563ce
I20260929 15:57:35.059001 7279 ts_tablet_manager.cc:1939] T aa223136e22145b9b802ab9c22e563ce P f2d0da34e8d442288d534be261b51db8: Deleting consensus metadata
I20260929 15:57:35.060856 6699 catalog_manager.cc:4841] TS f2d0da34e8d442288d534be261b51db8 (127.5.117.1:45749): tablet aa223136e22145b9b802ab9c22e563ce (table test-table [id=6d5660968f574e7ba3ef3a8ec2c0512e]) successfully deleted
I20260929 15:57:35.082108 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.058277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.5.117.62:34233)
0929 15:57:35.058619 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.058629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.066514 (+ 7885us) server_negotiation.cc:407] Connection header received
0929 15:57:35.066899 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.066906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.067115 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.067250 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.069024 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.069968 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.071473 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.071958 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.072133 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.075150 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.075157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.075161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.075212 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.078712 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.079674 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.079679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.079682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.079854 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.080376 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.080381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.080383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.080727 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.081014 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.081397 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.081704 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260929 15:57:35.082998 6714 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54754
I20260929 15:57:35.085459 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6786
I20260929 15:57:35.106509 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.087668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.5.117.1:45749)
0929 15:57:35.087930 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.087938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.087965 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:35.088206 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.088213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.088313 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.088434 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.090105 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.090964 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.092616 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.093320 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.093431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.097434 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.097443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.097450 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.097540 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.102501 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.103715 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.103723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.103727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.103988 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.104549 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.104556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.104560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.105031 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.105329 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.105726 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.106063 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 15:57:35.107642 6917 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53018
W20260929 15:57:35.108525 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6786: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:35.108798 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6786
I20260929 15:57:35.132419 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6926
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:35.148547 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.133465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.5.117.2:36003)
0929 15:57:35.133693 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.133698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.133718 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:57:35.133905 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.133910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.133984 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.134095 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.135323 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.136346 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.137609 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.138043 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.138144 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.142559 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.142573 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.142592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.142630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.145866 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.146546 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.146554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.146557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.146689 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.147028 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.147031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.147033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.147425 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.147673 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.147908 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.148163 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 15:57:35.149263 7056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51674
W20260929 15:57:35.149957 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6926: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:35.150131 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6926
I20260929 15:57:35.170727 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7066
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[6670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:35.187652 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.171632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.5.117.3:43277)
0929 15:57:35.171864 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.171871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.171973 (+ 102us) server_negotiation.cc:407] Connection header received
0929 15:57:35.172274 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.172279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.172376 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.172493 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.173896 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.174696 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.175996 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.176639 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.176786 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.181341 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.181357 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.181379 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.181418 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.184504 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.185272 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.185281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.185283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.185466 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.185863 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.185870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.185873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.186314 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.186746 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.186975 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.187244 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 15:57:35.188728 7197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50956
W20260929 15:57:35.189522 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:35.189719 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 7066
I20260929 15:57:35.214643 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 6683
I20260929 15:57:35.216240 6755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:35.349704 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 6683
2026-09-29T15:57:35Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4893 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[7289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[7289](info): set up 2 sockets
Sep 29 15:57:35 dist-test-slave-b69v krb5kdc[7289](info): commencing operation
krb5kdc: starting...
W20260929 15:57:36.480897 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:36 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:36.676538 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35293
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:37.041481 7302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:37.041852 7302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:37.042023 7302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:37.049373 7302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:37.049520 7302 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:57:37.049643 7302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:37.049767 7302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:37.049855 7302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:37.049974 7302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:37.061040 7302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:37.062610 7302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:37.064258 7302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:37.077512 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:37.077694 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:37.077649 7310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:37.078564 7302 server_base.cc:1034] running on GCE node
I20260929 15:57:37.079399 7302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:37.081014 7302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:37.082263 7302 hybrid_clock.cc:648] HybridClock initialized: now 1790697457082197 us; error 48 us; skew 500 ppm
I20260929 15:57:37.082617 7302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:37 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:37.086028 7302 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:37.087689 7302 webserver.cc:460] Webserver started at http://127.5.117.62:45885/ using document root <none> and password file <none>
I20260929 15:57:37.088577 7302 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:37.088714 7302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:37.089071 7302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:37.092430 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "0669a728c12c4f56a20f3301fe48eaab"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "6c1e224517d999bdffc406ef9b171f5a"
server_key_iv: "2046a4a34deb0ce64cb0b4605494cea6"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.093300 7302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "0669a728c12c4f56a20f3301fe48eaab"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "6c1e224517d999bdffc406ef9b171f5a"
server_key_iv: "2046a4a34deb0ce64cb0b4605494cea6"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.099026 7302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:37.102797 7317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:37.104282 7302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:37.104473 7302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "0669a728c12c4f56a20f3301fe48eaab"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "6c1e224517d999bdffc406ef9b171f5a"
server_key_iv: "2046a4a34deb0ce64cb0b4605494cea6"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.104723 7302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:37.117913 7302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:37.122584 7302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:37.122891 7302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:37.147320 7302 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:35293
I20260929 15:57:37.147328 7378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:35293 every 8 connection(s)
I20260929 15:57:37.149106 7302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:37.153862 7379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:37.158767 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 7302
I20260929 15:57:37.159034 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:37.159780 5588 external_mini_cluster.cc:1428] Setting key 4634086f3df3b397d5ee2cc5b13d3570
I20260929 15:57:37.166039 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Bootstrap starting.
I20260929 15:57:37.170291 7379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:37.171649 7379 log.cc:824] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:37.175602 7379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: No bootstrap required, opened a new log
Sep 29 15:57:37 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:37.183905 7379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } }
I20260929 15:57:37.184479 7379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:37.184612 7379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0669a728c12c4f56a20f3301fe48eaab, State: Initialized, Role: FOLLOWER
I20260929 15:57:37.185395 7379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } }
I20260929 15:57:37.185539 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:37.162364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.5.117.62:35293)
0929 15:57:37.163292 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:37.163309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:37.163358 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:57:37.165262 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:37.165286 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:37.165715 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:37.166045 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:37.167992 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.169448 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.171228 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.172008 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.172245 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:37.177377 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:37.177388 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:37.177408 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:37.177452 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:37.180959 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.181705 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.181718 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.181724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.181889 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.182275 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.182280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.182283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.182870 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:37.183235 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:37.183691 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 15:57:37.184208 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":403,"threads_started":1}
I20260929 15:57:37.185702 7379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:37.185886 7379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:37.186088 7379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:37.189141 7379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } }
I20260929 15:57:37.189668 7379 leader_election.cc:302] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0669a728c12c4f56a20f3301fe48eaab; no voters:
I20260929 15:57:37.190248 7379 leader_election.cc:288] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:37.190383 7384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:37.191125 7384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [term 1 LEADER]: Becoming Leader. State: Replica: 0669a728c12c4f56a20f3301fe48eaab, State: Running, Role: LEADER
I20260929 15:57:37.191716 7384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } }
I20260929 15:57:37.192581 7379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:37.194628 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0669a728c12c4f56a20f3301fe48eaab. Latest consensus state: current_term: 1 leader_uuid: "0669a728c12c4f56a20f3301fe48eaab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } } }
I20260929 15:57:37.194888 7385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0669a728c12c4f56a20f3301fe48eaab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0669a728c12c4f56a20f3301fe48eaab" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35293 } } }
I20260929 15:57:37.195369 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:37.195377 7385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:37.200143 7389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:37.206067 7389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:37.216367 7389 catalog_manager.cc:1347] Generated new cluster ID: adb267f2027b405d947a273d8c8f5f70
I20260929 15:57:37.216503 7389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:37.229280 7389 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:37.231403 7389 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:37.241997 7389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Generated new TSK 0
I20260929 15:57:37.243218 7389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:37.308282 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35293
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:37.686352 7405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:37.686714 7405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:37.686829 7405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:37.695070 7405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:37.695210 7405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:37.695356 7405 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:37.695454 7405 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:37.695612 7405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:37.706629 7405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:37.708321 7405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:37.710114 7405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:37.725633 7411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:37.725706 7410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:37.725610 7413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:37.731616 7405 server_base.cc:1034] running on GCE node
I20260929 15:57:37.732328 7405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:37.733327 7405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:37.734681 7405 hybrid_clock.cc:648] HybridClock initialized: now 1790697457734578 us; error 44 us; skew 500 ppm
I20260929 15:57:37.735316 7405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:37 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:37.739085 7405 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:37.740988 7405 webserver.cc:460] Webserver started at http://127.5.117.1:37169/ using document root <none> and password file <none>
I20260929 15:57:37.741788 7405 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:37.741943 7405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:37.742357 7405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:37.745693 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "4f7b534732ccf8a5be8f63a46a6ab5b6"
server_key_iv: "762545bdcbc8f0b30e8ba0de6ed71046"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.746523 7405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "4f7b534732ccf8a5be8f63a46a6ab5b6"
server_key_iv: "762545bdcbc8f0b30e8ba0de6ed71046"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.752513 7405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:57:37.756266 7420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:37.757679 7405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:37.757871 7405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
format_stamp: "Formatted at 2026-09-29 15:57:37 on dist-test-slave-b69v"
server_key: "4f7b534732ccf8a5be8f63a46a6ab5b6"
server_key_iv: "762545bdcbc8f0b30e8ba0de6ed71046"
server_key_version: "encryptionkey@0"
I20260929 15:57:37.758085 7405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:37.773537 7405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:37.777422 7405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:37.777709 7405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:37.780539 7405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:37.780661 7405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:37.780822 7405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:37.780932 7405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:37 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:37.811681 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:37.781236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:56307 (local address 127.5.117.62:35293)
0929 15:57:37.781912 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:37.781916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:37.782851 (+ 935us) server_negotiation.cc:407] Connection header received
0929 15:57:37.785113 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:37.785121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:37.785225 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:37.785358 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:37.787411 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.788762 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.791346 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.791793 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.791935 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:37.797421 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:37.797426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:37.797429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:37.797463 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:37.808066 (+ 10603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.809093 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.809101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.809104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.809307 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.809856 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.809861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.809865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.810153 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:37.810316 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:37.811044 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 15:57:37.811280 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":440,"threads_started":1}
I20260929 15:57:37.812952 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:37.781833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.1:56307)
0929 15:57:37.782691 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:37.782728 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:37.784820 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:37.785599 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:37.785606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:37.786333 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:37.787184 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:37.787207 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.788965 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.788972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:37.791188 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:37.791200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.792086 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.792094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:37.792356 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:37.793174 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:37.793222 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:37.797136 (+ 3914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:37.808219 (+ 11083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:37.808235 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:37.808249 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:37.808966 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:37.809494 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:37.809501 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:37.809506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:37.809726 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:37.810282 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:37.810290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:37.810665 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 15:57:37.810948 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 15:57:37.811474 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":381,"threads_started":1}
I20260929 15:57:37.830902 7405 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:40021
I20260929 15:57:37.830963 7540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:40021 every 8 connection(s)
I20260929 15:57:37.832397 7405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:37.834932 7541 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35293
I20260929 15:57:37.835148 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:37.820242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.1:51967)
0929 15:57:37.821737 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:37.821759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:37.821871 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:37.822225 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:37.822228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:37.822619 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:37.822899 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:37.822907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.824659 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.824664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:37.826358 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:37.826370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.827339 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.827345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:37.827493 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:37.828200 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:37.828220 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:37.829182 (+ 962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:37.832417 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:37.832422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:37.832426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:37.832851 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:37.833258 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:37.833264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:37.833267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:37.833400 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:37.833933 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:37.833938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:37.834135 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 15:57:37.834287 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 15:57:37.834632 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"thread_start_us":1084,"threads_started":1}
I20260929 15:57:37.835244 7541 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:37.835799 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:37.820631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:51967 (local address 127.5.117.62:35293)
0929 15:57:37.820844 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:37.820848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:37.821856 (+ 1008us) server_negotiation.cc:407] Connection header received
0929 15:57:37.821970 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:37.821978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:37.822088 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:37.822214 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:37.823041 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.824496 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.826497 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:37.827145 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:37.827287 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:37.829367 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:37.829372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:37.829375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:37.829403 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:37.832290 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.833001 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.833009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.833014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.833144 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:37.833521 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:37.833541 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:37.833543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:37.833806 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:37.833967 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:37.835117 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0929 15:57:37.835529 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:57:37.835943 7541 heartbeater.cc:511] Master 127.5.117.62:35293 requested a full tablet report, sending...
I20260929 15:57:37.838485 7334 ts_manager.cc:194] Registered new tserver with Master: 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021)
I20260929 15:57:37.840319 7334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:56307
I20260929 15:57:37.841483 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 7405
I20260929 15:57:37.841694 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:37.842569 5588 external_mini_cluster.cc:1428] Setting key 6551796d18e6d28f94a5498e40409f9c
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:37.883185 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35293
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:38.236990 7545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:38.237329 7545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:38.237506 7545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:38.244338 7545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:38.244475 7545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:38.244640 7545 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:38.244729 7545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:38.244844 7545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:38.253916 7545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:38.255436 7545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:38.256958 7545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:38.270718 7553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.270792 7551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.270812 7550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:38.271818 7545 server_base.cc:1034] running on GCE node
I20260929 15:57:38.272405 7545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:38.273351 7545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:38.274575 7545 hybrid_clock.cc:648] HybridClock initialized: now 1790697458274527 us; error 91 us; skew 500 ppm
I20260929 15:57:38.274921 7545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:38.278275 7545 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:38.279654 7545 webserver.cc:460] Webserver started at http://127.5.117.2:34435/ using document root <none> and password file <none>
I20260929 15:57:38.280361 7545 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:38.280458 7545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:38.280714 7545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:38.283581 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "660858a14c774bd5a44400bd02ec265a"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "5dff7451655ea30d265a765aae275d80"
server_key_iv: "05bcf313c98c0813d3a2ff65cb724c11"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.284412 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "660858a14c774bd5a44400bd02ec265a"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "5dff7451655ea30d265a765aae275d80"
server_key_iv: "05bcf313c98c0813d3a2ff65cb724c11"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.289414 7545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 15:57:38.292662 7560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:38.294240 7545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 15:57:38.294446 7545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "660858a14c774bd5a44400bd02ec265a"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "5dff7451655ea30d265a765aae275d80"
server_key_iv: "05bcf313c98c0813d3a2ff65cb724c11"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.294651 7545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:38.319885 7545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:38.323781 7545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:38.324088 7545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:38.326659 7545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:38.326774 7545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:38.326910 7545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:38.327008 7545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:38.352793 7569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.327991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.2:37017)
0929 15:57:38.329005 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:38.329057 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:38.330966 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:38.331782 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:38.331790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:38.332560 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:38.333468 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.333522 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.335372 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.335384 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.337679 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.337703 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.338590 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.338603 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.338892 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.339685 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:38.339705 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:38.342979 (+ 3274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:38.347303 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.347313 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.347335 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.347983 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.348530 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.348537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.348544 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.348943 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.349677 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:38.349687 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:38.350210 (+ 523us) client_negotiation.cc:769] Sending connection context
0929 15:57:38.350536 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 15:57:38.351063 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":429,"threads_started":1}
I20260929 15:57:38.352883 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.327445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:37017 (local address 127.5.117.62:35293)
0929 15:57:38.327671 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.327678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.329195 (+ 1517us) server_negotiation.cc:407] Connection header received
0929 15:57:38.331247 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.331256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.331373 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.331501 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.333721 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.335143 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.337863 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.338306 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.338435 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.343322 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.343330 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.343335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.343386 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.347166 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.348192 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.348199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.348202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.348372 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.349076 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.349082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.349086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.349519 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.349709 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.352264 (+ 2555us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.352510 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:57:38.373009 7545 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:39007
I20260929 15:57:38.373085 7679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:39007 every 8 connection(s)
I20260929 15:57:38.374650 7545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:38.377206 7680 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35293
I20260929 15:57:38.377540 7680 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:38.377827 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.361227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.2:52955)
0929 15:57:38.362313 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:38.362335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:38.362514 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:38.362869 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:38.362874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:38.363384 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:38.363693 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.363703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.365138 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.365144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.366874 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.366889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.367962 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.367975 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.368247 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.369356 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:38.369400 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:38.370906 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:38.374305 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.374311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.374315 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.374955 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.375416 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.375421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.375429 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.375641 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.376633 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:38.376643 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:38.376846 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 15:57:38.377016 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 15:57:38.377337 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":799,"threads_started":1}
I20260929 15:57:38.377847 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.361178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:52955 (local address 127.5.117.62:35293)
0929 15:57:38.361393 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.361401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.362436 (+ 1035us) server_negotiation.cc:407] Connection header received
0929 15:57:38.362659 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.362666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.362741 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.362842 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.363839 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.364992 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.367103 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.367704 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.367852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.371087 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.371092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.371095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.371124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.374168 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.375076 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.375080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.375082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.375253 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.375847 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.375853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.375856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.376453 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.376640 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.377155 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.377440 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:38.378589 7680 heartbeater.cc:511] Master 127.5.117.62:35293 requested a full tablet report, sending...
I20260929 15:57:38.379741 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 7545
I20260929 15:57:38.379972 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:38.380359 7334 ts_manager.cc:194] Registered new tserver with Master: 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007)
I20260929 15:57:38.380641 5588 external_mini_cluster.cc:1428] Setting key 77d55e7b4f7489270c705c70840d77aa
I20260929 15:57:38.381536 7334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:37017
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:38.424623 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35293
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:38.779201 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:38.779498 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:38.779681 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:38.786638 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:38.786742 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:38.786849 7684 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:38.786928 7684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:38.787062 7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:38.797410 7684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:38.799101 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:38.800722 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:38.816608 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.816409 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.816464 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:38.818118 7684 server_base.cc:1034] running on GCE node
I20260929 15:57:38.818796 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:38.819803 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:38.821058 7684 hybrid_clock.cc:648] HybridClock initialized: now 1790697458820995 us; error 86 us; skew 500 ppm
I20260929 15:57:38.821401 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:38.825429 7684 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:38.827653 7684 webserver.cc:460] Webserver started at http://127.5.117.3:33327/ using document root <none> and password file <none>
I20260929 15:57:38.828681 7684 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:38.828867 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:38.829242 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:38.832705 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "2548f375f8ae58bcd0ecd327beb915c2"
server_key_iv: "cfff3d28b40b25bcf16743334153fd44"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.833566 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "2548f375f8ae58bcd0ecd327beb915c2"
server_key_iv: "cfff3d28b40b25bcf16743334153fd44"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.839345 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:38.843277 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:38.843588 7541 heartbeater.cc:503] Master 127.5.117.62:35293 was elected leader, sending a full tablet report...
I20260929 15:57:38.844915 7684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:38.845111 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-b69v"
server_key: "2548f375f8ae58bcd0ecd327beb915c2"
server_key_iv: "cfff3d28b40b25bcf16743334153fd44"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.845341 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:38.859790 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:38.864380 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:38.864693 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:38.867550 7684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:38.867695 7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:38.867846 7684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:38.867945 7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:38.894922 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.869042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.3:37019)
0929 15:57:38.869999 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:38.870052 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:38.872498 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:38.873327 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:38.873344 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:38.874168 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:38.875034 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.875056 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.877130 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.877136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.879479 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.879497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.880624 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.880632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.880849 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.881603 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:38.881624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:38.885009 (+ 3385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:38.889172 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.889180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.889194 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.889878 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.890303 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.890308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.890312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.890646 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.891476 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:38.891487 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:38.891983 (+ 496us) client_negotiation.cc:769] Sending connection context
0929 15:57:38.892390 (+ 407us) client_negotiation.cc:240] Negotiation successful
0929 15:57:38.893042 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":448,"threads_started":1}
I20260929 15:57:38.895685 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.868484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:37019 (local address 127.5.117.62:35293)
0929 15:57:38.868683 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.868691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.870184 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 15:57:38.872818 (+ 2634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.872824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.872948 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.873071 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.875308 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.876585 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.879655 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.880373 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.880509 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.885262 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.885268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.885271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.885306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.889006 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.890009 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.890017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.890021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.890174 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.890821 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.890828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.890833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.891212 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.891376 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.895022 (+ 3646us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.895294 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:38.916617 7684 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34839
I20260929 15:57:38.916646 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34839 every 8 connection(s)
I20260929 15:57:38.918190 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:38.920532 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.903934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.5.117.3:35415)
0929 15:57:38.904816 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:38.904846 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:38.905096 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:38.905488 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:38.905492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:38.905985 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:38.906406 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.906418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.907778 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.907785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.909614 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:38.909640 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.911030 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.911036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:38.911161 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.912261 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:38.912280 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:38.913534 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:38.917475 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.917483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.917487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.917897 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.918287 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:38.918294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:38.918298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:38.918429 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:38.919229 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:38.919237 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:38.919467 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 15:57:38.919692 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 15:57:38.920023 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":569,"threads_started":1}
I20260929 15:57:38.921012 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.903941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:35415 (local address 127.5.117.62:35293)
0929 15:57:38.904185 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.904194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.904956 (+ 762us) server_negotiation.cc:407] Connection header received
0929 15:57:38.905259 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.905264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.905350 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.905460 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.906556 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.907645 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.909838 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.910569 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.910730 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.913830 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.913835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.913839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.913871 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.917239 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.918033 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.918046 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.918048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.918173 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.918550 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.918556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.918559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.918996 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.919164 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.920256 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.920546 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:57:38.921080 7819 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35293
I20260929 15:57:38.921614 7819 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:38.922499 7819 heartbeater.cc:511] Master 127.5.117.62:35293 requested a full tablet report, sending...
I20260929 15:57:38.922830 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 7684
I20260929 15:57:38.923062 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:38.923696 5588 external_mini_cluster.cc:1428] Setting key 0f62d95fd2847296fac6f90d94933fe8
I20260929 15:57:38.924110 7333 ts_manager.cc:194] Registered new tserver with Master: ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:38.925446 7333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:37019
I20260929 15:57:38.927122 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:38 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:38.981148 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.964010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.5.117.62:35293)
0929 15:57:38.964332 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.964341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.964707 (+ 366us) server_negotiation.cc:407] Connection header received
0929 15:57:38.964929 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.964934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.965011 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.965116 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.967072 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.967901 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.969537 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.970348 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.970549 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.975075 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.975080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.975083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.975118 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.977841 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.978782 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.978789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.978793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.978990 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.979441 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.979447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.979450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.979759 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.980043 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.980411 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.980691 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 15:57:38.985957 7333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:38.993701 7333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:39.025805 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.007698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.5.117.3:34839)
0929 15:57:39.008703 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.008714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.016994 (+ 8280us) server_negotiation.cc:407] Connection header received
0929 15:57:39.017163 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.017178 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.017572 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.017835 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.018743 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.020693 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.023105 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.024550 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.024709 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.024989 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.025217 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.025399 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":560,"threads_started":1}
I20260929 15:57:39.025889 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.007432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34839 (local address 127.0.0.1:54122)
0929 15:57:39.013513 (+ 6081us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.013551 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.013740 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.017836 (+ 4096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.017844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.017875 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.018382 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.018411 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.020836 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.020842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.022924 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.022937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.024738 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.024744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.024936 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.024994 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.025186 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.025400 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5809,"mutex_wait_us":24,"thread_start_us":5672,"threads_started":1}
I20260929 15:57:39.027135 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.007539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.0.0.1:57472)
0929 15:57:39.016664 (+ 9125us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.016690 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.016863 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.018213 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.018220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.018249 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.019251 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.019268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.021125 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.021133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.023675 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.023690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.025862 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.025871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.026400 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.026433 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.026534 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.026683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8931,"mutex_wait_us":79,"thread_start_us":8718,"threads_started":1}
I20260929 15:57:39.027158 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.008694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.5.117.2:39007)
0929 15:57:39.017303 (+ 8609us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.017315 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.017354 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:57:39.017492 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.017501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.017765 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.017988 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.019502 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.020985 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.023863 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.025348 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.025482 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.025682 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.026646 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.026824 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8435,"thread_start_us":8267,"threads_started":1}
I20260929 15:57:39.027765 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.008420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.5.117.1:40021)
0929 15:57:39.014806 (+ 6386us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.014813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.017967 (+ 3154us) server_negotiation.cc:407] Connection header received
0929 15:57:39.018109 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.018117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.018319 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.018523 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.019910 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.021541 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.024210 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.025747 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.025890 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.026080 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.027243 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.027421 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6215,"thread_start_us":6065,"threads_started":1}
I20260929 15:57:39.028308 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.008752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40021 (local address 127.0.0.1:42568)
0929 15:57:39.017673 (+ 8921us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.017693 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.017828 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.018518 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.018523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.018548 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.019441 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.019468 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.021706 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.021711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.024010 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.024022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.026527 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.026547 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.026725 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.026756 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.026851 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.026984 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8811,"mutex_wait_us":22,"thread_start_us":8661,"threads_started":1}
I20260929 15:57:39.030658 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 (DEFAULT_TABLE table=test-table [id=22608006576b4a8995d60716a5df21c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.030794 7610 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 (DEFAULT_TABLE table=test-table [id=22608006576b4a8995d60716a5df21c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.032205 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8837de3e9c46109b28560eb1d43e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.032222 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8837de3e9c46109b28560eb1d43e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.034730 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 (DEFAULT_TABLE table=test-table [id=22608006576b4a8995d60716a5df21c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.036036 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8837de3e9c46109b28560eb1d43e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.044375 7838 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Bootstrap starting.
I20260929 15:57:39.047127 7839 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Bootstrap starting.
I20260929 15:57:39.048751 7838 tablet_bootstrap.cc:654] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.050128 7838 log.cc:824] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:39.050849 7839 tablet_bootstrap.cc:654] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.050899 7840 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Bootstrap starting.
I20260929 15:57:39.052274 7839 log.cc:824] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:39.054611 7838 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: No bootstrap required, opened a new log
I20260929 15:57:39.054899 7838 ts_tablet_manager.cc:1397] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 15:57:39.054949 7840 tablet_bootstrap.cc:654] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.055284 7839 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: No bootstrap required, opened a new log
I20260929 15:57:39.055552 7839 ts_tablet_manager.cc:1397] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:57:39.056239 7840 log.cc:824] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:39.058589 7840 tablet_bootstrap.cc:492] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: No bootstrap required, opened a new log
I20260929 15:57:39.058852 7840 ts_tablet_manager.cc:1397] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:57:39.066484 7839 raft_consensus.cc:348] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.066478 7838 raft_consensus.cc:348] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.066920 7838 raft_consensus.cc:374] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.066920 7839 raft_consensus.cc:374] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.067028 7838 raft_consensus.cc:729] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.067028 7839 raft_consensus.cc:729] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.067099 7840 raft_consensus.cc:348] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.067540 7840 raft_consensus.cc:374] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.067654 7840 raft_consensus.cc:729] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.067669 7839 consensus_queue.cc:260] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.067669 7838 consensus_queue.cc:260] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.068296 7840 consensus_queue.cc:260] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.071089 7840 ts_tablet_manager.cc:1428] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 15:57:39.074947 7680 heartbeater.cc:503] Master 127.5.117.62:35293 was elected leader, sending a full tablet report...
I20260929 15:57:39.075011 7838 ts_tablet_manager.cc:1428] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260929 15:57:39.075506 7839 ts_tablet_manager.cc:1428] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Time spent starting tablet: real 0.020s user 0.017s sys 0.004s
I20260929 15:57:39.076371 7819 heartbeater.cc:503] Master 127.5.117.62:35293 was elected leader, sending a full tablet report...
W20260929 15:57:39.086901 7542 tablet.cc:2369] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:39.127271 7681 tablet.cc:2369] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:39.170183 7820 tablet.cc:2369] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:39.304642 7846 raft_consensus.cc:482] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:39.305033 7846 raft_consensus.cc:504] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.307168 7846 leader_election.cc:288] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839), 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007)
I20260929 15:57:39.318833 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.307934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:44263 (local address 127.5.117.3:34839)
0929 15:57:39.308275 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.308287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.309315 (+ 1028us) server_negotiation.cc:407] Connection header received
0929 15:57:39.309726 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.309735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.309841 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.309920 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.311190 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.312445 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.315848 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.317259 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.317434 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.317578 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.318141 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.318299 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 15:57:39.318828 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.308113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:57013 (local address 127.5.117.2:39007)
0929 15:57:39.308404 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.308413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.309857 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 15:57:39.309972 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.309976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.310087 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.310179 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.311191 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.312453 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.315869 (+ 3416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.317292 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.317439 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.317578 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.318142 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.318298 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 15:57:39.318881 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.307441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34839 (local address 127.5.117.1:44263)
0929 15:57:39.309161 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.309186 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.309477 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.310224 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.310232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.310342 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.310970 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.310986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.312639 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.312647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.315614 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.315635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.317652 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.317657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.317904 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.317969 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.318093 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.318314 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":231,"spinlock_wait_cycles":36096,"thread_start_us":1237,"threads_started":1}
I20260929 15:57:39.318928 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.307611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.5.117.1:57013)
0929 15:57:39.309711 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.309728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.309856 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.310224 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.310232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.310271 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.310976 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.310986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.312640 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.312647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.315614 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.315635 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.317635 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.317644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.317909 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.317969 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.318104 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.318314 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1981,"mutex_wait_us":340,"thread_start_us":1734,"threads_started":1}
I20260929 15:57:39.321038 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8837de3e9c46109b28560eb1d43e42" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a" is_pre_election: true
I20260929 15:57:39.321043 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8837de3e9c46109b28560eb1d43e42" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" is_pre_election: true
I20260929 15:57:39.322028 7630 raft_consensus.cc:2454] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 0.
I20260929 15:57:39.322028 7769 raft_consensus.cc:2454] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 0.
I20260929 15:57:39.323050 7422 leader_election.cc:302] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:39.323601 7846 raft_consensus.cc:2781] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:39.323817 7846 raft_consensus.cc:482] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:39.323954 7846 raft_consensus.cc:3037] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.326630 7846 raft_consensus.cc:504] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.327417 7846 leader_election.cc:288] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 election: Requested vote from peers ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839), 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007)
I20260929 15:57:39.328414 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8837de3e9c46109b28560eb1d43e42" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
I20260929 15:57:39.328473 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8837de3e9c46109b28560eb1d43e42" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a"
I20260929 15:57:39.328737 7769 raft_consensus.cc:3037] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.328738 7630 raft_consensus.cc:3037] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.332711 7630 raft_consensus.cc:2454] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 1.
I20260929 15:57:39.332715 7769 raft_consensus.cc:2454] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 1.
I20260929 15:57:39.333403 7424 leader_election.cc:302] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, 660858a14c774bd5a44400bd02ec265a; no voters:
I20260929 15:57:39.333792 7846 raft_consensus.cc:2781] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:39.334378 7846 raft_consensus.cc:686] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 LEADER]: Becoming Leader. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Running, Role: LEADER
I20260929 15:57:39.334976 7846 consensus_queue.cc:237] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } }
I20260929 15:57:39.341041 7334 catalog_manager.cc:5495] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1). New cstate: current_term: 1 leader_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } health_report { overall_health: HEALTHY } } }
Sep 29 15:57:39 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:39.432276 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.415437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35293 (local address 127.0.0.1:33612)
0929 15:57:39.416423 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.416442 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.416617 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.417054 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.417059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.417915 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:39.418251 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.418262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.419503 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.419510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.420917 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.420936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.421845 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.421850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.421997 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.422941 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:39.422962 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:39.425975 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:39.429625 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:39.429639 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:39.429646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:39.430093 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:39.430511 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:39.430514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:39.430517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:39.430662 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:39.431217 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:39.431225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:39.431384 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.431528 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.431798 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":542,"threads_started":1}
I20260929 15:57:39.432505 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.415437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.5.117.62:35293)
0929 15:57:39.415684 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.415698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.416567 (+ 869us) server_negotiation.cc:407] Connection header received
0929 15:57:39.416751 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.416755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.416835 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.416948 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:39.418433 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.419279 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.421117 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.421610 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.421725 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.426216 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:39.426223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:39.426228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:39.426268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:39.429432 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:39.430254 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:39.430259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:39.430261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:39.430383 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:39.430765 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:39.430769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:39.430771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:39.431079 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:39.431196 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.431670 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.432124 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":83}
I20260929 15:57:39.439610 7334 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:33612:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:57:39.440970 7334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:39.449671 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 11b67186284145fcb276303f281dc43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=924d533bb932407885ce1d74776c69cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:39.449582 7610 tablet_service.cc:1467] Processing CreateTablet for tablet 11b67186284145fcb276303f281dc43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=924d533bb932407885ce1d74776c69cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:39.449632 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 11b67186284145fcb276303f281dc43f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=924d533bb932407885ce1d74776c69cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:39.450790 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b67186284145fcb276303f281dc43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.450892 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b67186284145fcb276303f281dc43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.450932 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b67186284145fcb276303f281dc43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.457682 7840 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Bootstrap starting.
I20260929 15:57:39.458612 7839 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: Bootstrap starting.
I20260929 15:57:39.458614 7838 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: Bootstrap starting.
I20260929 15:57:39.459643 7840 tablet_bootstrap.cc:654] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.460484 7839 tablet_bootstrap.cc:654] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.460484 7838 tablet_bootstrap.cc:654] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.462949 7840 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: No bootstrap required, opened a new log
I20260929 15:57:39.463116 7840 ts_tablet_manager.cc:1397] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 15:57:39.463454 7838 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: No bootstrap required, opened a new log
I20260929 15:57:39.463482 7839 tablet_bootstrap.cc:492] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: No bootstrap required, opened a new log
I20260929 15:57:39.463625 7838 ts_tablet_manager.cc:1397] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:57:39.463624 7839 ts_tablet_manager.cc:1397] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:39.463924 7840 raft_consensus.cc:348] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.464223 7840 raft_consensus.cc:374] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.464329 7840 raft_consensus.cc:729] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.464455 7839 raft_consensus.cc:348] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.464448 7838 raft_consensus.cc:348] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.464597 7840 consensus_queue.cc:260] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.464731 7838 raft_consensus.cc:374] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.464730 7839 raft_consensus.cc:374] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.464833 7838 raft_consensus.cc:729] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.464833 7839 raft_consensus.cc:729] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.465068 7838 consensus_queue.cc:260] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.465209 7840 ts_tablet_manager.cc:1428] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:39.465085 7839 consensus_queue.cc:260] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.465682 7838 ts_tablet_manager.cc:1428] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:57:39.465605 7845 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.466022 7845 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.466519 7846 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.466703 7839 ts_tablet_manager.cc:1428] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:39.466596 7844 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.466796 7846 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.466899 7844 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.684012 7845 raft_consensus.cc:482] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:39.684331 7845 raft_consensus.cc:504] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.686223 7845 leader_election.cc:288] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:39.693907 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.686694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:45939 (local address 127.5.117.3:34839)
0929 15:57:39.686931 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.686942 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.687654 (+ 712us) server_negotiation.cc:407] Connection header received
0929 15:57:39.688096 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.688107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.688229 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.688288 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.689059 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.689835 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.691920 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.692817 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.693002 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.693135 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.693453 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.693583 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:57:39.693998 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.686692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36643 (local address 127.5.117.1:40021)
0929 15:57:39.688158 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.688166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.688188 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:57:39.688276 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.688280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.688341 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.688450 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:39.689209 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.690005 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.691908 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.692853 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.693042 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.693169 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.693486 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.693656 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1322,"thread_start_us":1142,"threads_started":1}
I20260929 15:57:39.694160 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.686518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34839 (local address 127.5.117.2:45939)
0929 15:57:39.687492 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.687531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.687852 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.688467 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.688470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.688500 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.688884 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.688894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.689964 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.689971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.691772 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.691783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.693100 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.693108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.693266 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.693311 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.693424 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.693580 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":11,"spinlock_wait_cycles":2816,"thread_start_us":979,"threads_started":1}
I20260929 15:57:39.694185 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.686518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40021 (local address 127.5.117.2:36643)
0929 15:57:39.687872 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.687892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.688105 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.688467 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.688472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.688518 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:39.689019 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.689028 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.690132 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.690141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.691761 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.691772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.693144 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.693153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.693325 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.693347 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.693454 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.693581 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":115,"spinlock_wait_cycles":6400,"thread_start_us":521,"threads_started":1}
I20260929 15:57:39.694941 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b67186284145fcb276303f281dc43f" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" is_pre_election: true
I20260929 15:57:39.695232 7769 raft_consensus.cc:2454] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 0.
I20260929 15:57:39.695192 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b67186284145fcb276303f281dc43f" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" is_pre_election: true
I20260929 15:57:39.695657 7491 raft_consensus.cc:2454] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 0.
I20260929 15:57:39.695771 7562 leader_election.cc:302] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 660858a14c774bd5a44400bd02ec265a, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:39.696259 7845 raft_consensus.cc:2781] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:39.696391 7845 raft_consensus.cc:482] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:39.696502 7845 raft_consensus.cc:3037] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.699082 7845 raft_consensus.cc:504] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.699657 7845 leader_election.cc:288] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 election: Requested vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:39.700165 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b67186284145fcb276303f281dc43f" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
I20260929 15:57:39.700172 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b67186284145fcb276303f281dc43f" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
I20260929 15:57:39.700414 7769 raft_consensus.cc:3037] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.700393 7491 raft_consensus.cc:3037] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:39.703861 7769 raft_consensus.cc:2454] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 1.
I20260929 15:57:39.703874 7491 raft_consensus.cc:2454] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 1.
I20260929 15:57:39.704535 7561 leader_election.cc:302] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, 660858a14c774bd5a44400bd02ec265a, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:39.705098 7845 raft_consensus.cc:2781] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:39.705883 7845 raft_consensus.cc:686] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [term 1 LEADER]: Becoming Leader. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Running, Role: LEADER
I20260929 15:57:39.706580 7845 consensus_queue.cc:237] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.710687 7863 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: TxnStatusTablet state changed. Reason: New leader 660858a14c774bd5a44400bd02ec265a. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.711155 7863 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:39.711880 7865 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:57:39.712193 7865 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:57:39.712656 7332 catalog_manager.cc:5495] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a reported cstate change: term changed from 0 to 1, leader changed from <none> to 660858a14c774bd5a44400bd02ec265a (127.5.117.2). New cstate: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:39.731129 7846 consensus_queue.cc:1035] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:39.739830 7870 consensus_queue.cc:1035] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:39.763662 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.756970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.0.0.1:57480)
0929 15:57:39.757231 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.757254 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.757442 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.757903 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.757908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.757921 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:39.758145 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.758155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.759414 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.759421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.760702 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.760713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.761849 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.761857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.762011 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.762095 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.763163 (+ 1068us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.763191 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.763268 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.763391 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:57:39.763772 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.757052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.5.117.2:39007)
0929 15:57:39.757225 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.757232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.757338 (+ 106us) server_negotiation.cc:407] Connection header received
0929 15:57:39.757646 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.757656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.757745 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.757885 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:39.758283 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.759270 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.760899 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.761640 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.761772 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.762312 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.763043 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.763192 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.763324 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.763420 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:57:39.770757 7769 raft_consensus.cc:1261] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Refusing update from remote peer 660858a14c774bd5a44400bd02ec265a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:39.771013 7491 raft_consensus.cc:1261] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Refusing update from remote peer 660858a14c774bd5a44400bd02ec265a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:39.771430 7863 consensus_queue.cc:1035] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:39.771862 7845 consensus_queue.cc:1035] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:39.778806 7844 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: TxnStatusTablet state changed. Reason: New leader 660858a14c774bd5a44400bd02ec265a. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.779171 7844 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.781919 7870 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: TxnStatusTablet state changed. Reason: New leader 660858a14c774bd5a44400bd02ec265a. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.782260 7870 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.783918 7872 mvcc.cc:205] Tried to move back new op lower bound from 7334696795212767232 to 7334696794962456576. Current Snapshot: MvccSnapshot[applied={T|T < 7334696795212767232}]
I20260929 15:57:39.784689 7869 mvcc.cc:205] Tried to move back new op lower bound from 7334696795212767232 to 7334696794962456576. Current Snapshot: MvccSnapshot[applied={T|T < 7334696795212767232}]
I20260929 15:57:39.792922 7844 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.793179 7844 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P ed6ddfc1e32b4c8ebf0fba373bbf7138: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.795825 7870 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.796115 7870 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 1487d31f6e3f4f09bc71d2c8bb91b5ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:39.796252 7845 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.796550 7845 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:39.797444 7863 tablet_replica.cc:439] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } } }
I20260929 15:57:39.797717 7863 tablet_replica.cc:442] T 11b67186284145fcb276303f281dc43f P 660858a14c774bd5a44400bd02ec265a: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:39.815794 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.808690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.5.117.1:40021)
0929 15:57:39.808915 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.808923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.809374 (+ 451us) server_negotiation.cc:407] Connection header received
0929 15:57:39.809635 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.809641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.809789 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.809956 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:39.810472 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.811718 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.813180 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.813839 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.813972 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.814333 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.815152 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.815309 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.815360 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.815454 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:57:39.835120 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.827923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:59111 (local address 127.5.117.2:39007)
0929 15:57:39.828165 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.828172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.828736 (+ 564us) server_negotiation.cc:407] Connection header received
0929 15:57:39.828970 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.828976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.829104 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.829202 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:39.829616 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.830849 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.832917 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.833668 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.833801 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.834304 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.834448 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.834567 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.834709 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.834805 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:57:39.835157 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.827888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.5.117.1:59111)
0929 15:57:39.828624 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.828647 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.828809 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.829233 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.829237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.829247 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:39.829493 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.829500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.831019 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.831026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.832764 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.832778 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.833834 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.833840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.834006 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.834079 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.834570 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.834582 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.834667 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.834822 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":408,"threads_started":1}
I20260929 15:57:39.883160 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.876420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:34025 (local address 127.5.117.1:40021)
0929 15:57:39.876628 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:39.876637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:39.877395 (+ 758us) server_negotiation.cc:407] Connection header received
0929 15:57:39.877705 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:39.877711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:39.877810 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:39.877870 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:39.878665 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.879403 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.881340 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.881960 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.882081 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.882462 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.882573 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.882659 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:39.882783 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 15:57:39.882857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:57:39.883262 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:39.876465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40021 (local address 127.5.117.2:34025)
0929 15:57:39.877261 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:39.877283 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:39.877538 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:39.878088 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:39.878098 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:39.878111 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:39.878499 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.878510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.879579 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.879586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.881209 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:39.881221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:39.882111 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:39.882117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:39.882251 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:39.882301 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:39.882666 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:39.882681 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 15:57:39.882754 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 15:57:39.882878 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":406,"threads_started":1}
I20260929 15:57:39.922310 7333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:39.922650 7333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:39.927169 7333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Sending DeleteTablet for 3 replicas of tablet 3e8837de3e9c46109b28560eb1d43e42
I20260929 15:57:39.928377 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54122
I20260929 15:57:39.929946 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42568
I20260929 15:57:39.933266 7610 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e8837de3e9c46109b28560eb1d43e42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:57472
I20260929 15:57:39.943990 7333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:39.945043 7333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:39.953675 7470 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 (DEFAULT_TABLE table=test-table [id=b725b31f2b5e4b76850b3495e838cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.954156 7609 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 (DEFAULT_TABLE table=test-table [id=b725b31f2b5e4b76850b3495e838cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.954381 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8b32a75c9f40b99a543e7b8d162a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.954474 7748 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 (DEFAULT_TABLE table=test-table [id=b725b31f2b5e4b76850b3495e838cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:39.954748 7609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8b32a75c9f40b99a543e7b8d162a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.955054 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8b32a75c9f40b99a543e7b8d162a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:39.961498 7840 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Bootstrap starting.
I20260929 15:57:39.961822 7839 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Bootstrap starting.
I20260929 15:57:39.963327 7840 tablet_bootstrap.cc:654] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.963486 7839 tablet_bootstrap.cc:654] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.964679 7838 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Bootstrap starting.
I20260929 15:57:39.966393 7838 tablet_bootstrap.cc:654] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:39.977152 7838 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: No bootstrap required, opened a new log
I20260929 15:57:39.977349 7838 ts_tablet_manager.cc:1397] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260929 15:57:39.977653 7840 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: No bootstrap required, opened a new log
I20260929 15:57:39.977854 7840 ts_tablet_manager.cc:1397] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260929 15:57:39.978279 7838 raft_consensus.cc:348] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.978549 7838 raft_consensus.cc:374] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.978634 7838 raft_consensus.cc:729] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.978760 7840 raft_consensus.cc:348] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.979067 7840 raft_consensus.cc:374] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.978994 7838 consensus_queue.cc:260] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.979166 7840 raft_consensus.cc:729] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.979450 7840 consensus_queue.cc:260] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.980202 7840 ts_tablet_manager.cc:1428] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 15:57:39.982911 7838 ts_tablet_manager.cc:1428] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260929 15:57:39.984302 7839 tablet_bootstrap.cc:492] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: No bootstrap required, opened a new log
I20260929 15:57:39.984527 7839 ts_tablet_manager.cc:1397] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
I20260929 15:57:39.985417 7839 raft_consensus.cc:348] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.985687 7839 raft_consensus.cc:374] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:39.985793 7839 raft_consensus.cc:729] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Initialized, Role: FOLLOWER
I20260929 15:57:39.986099 7839 consensus_queue.cc:260] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:39.987851 7839 ts_tablet_manager.cc:1428] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:40.000638 7894 tablet_replica.cc:330] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: stopping tablet replica
I20260929 15:57:40.001809 7893 tablet_replica.cc:330] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: stopping tablet replica
I20260929 15:57:40.002331 7893 raft_consensus.cc:2229] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:40.003119 7893 raft_consensus.cc:2258] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:40.003767 7894 raft_consensus.cc:2229] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:40.004315 7894 raft_consensus.cc:2258] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:40.007144 7894 ts_tablet_manager.cc:1905] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:40.010167 7892 tablet_replica.cc:330] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: stopping tablet replica
I20260929 15:57:40.010651 7892 raft_consensus.cc:2229] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:40.011073 7892 raft_consensus.cc:2258] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:40.011289 7893 ts_tablet_manager.cc:1905] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:40.013517 7892 ts_tablet_manager.cc:1905] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:40.021358 7892 ts_tablet_manager.cc:1918] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:40.021522 7892 log.cc:1196] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/3e8837de3e9c46109b28560eb1d43e42
I20260929 15:57:40.022070 7892 ts_tablet_manager.cc:1939] T 3e8837de3e9c46109b28560eb1d43e42 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting consensus metadata
I20260929 15:57:40.024238 7319 catalog_manager.cc:4841] TS ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839): tablet 3e8837de3e9c46109b28560eb1d43e42 (table test-table [id=22608006576b4a8995d60716a5df21c3]) successfully deleted
I20260929 15:57:40.024366 7894 ts_tablet_manager.cc:1918] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:40.024569 7894 log.cc:1196] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/3e8837de3e9c46109b28560eb1d43e42
I20260929 15:57:40.028311 7893 ts_tablet_manager.cc:1918] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:40.028525 7893 log.cc:1196] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/3e8837de3e9c46109b28560eb1d43e42
I20260929 15:57:40.028890 7894 ts_tablet_manager.cc:1939] T 3e8837de3e9c46109b28560eb1d43e42 P 660858a14c774bd5a44400bd02ec265a: Deleting consensus metadata
I20260929 15:57:40.030750 7893 ts_tablet_manager.cc:1939] T 3e8837de3e9c46109b28560eb1d43e42 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting consensus metadata
I20260929 15:57:40.031177 7321 catalog_manager.cc:4841] TS 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007): tablet 3e8837de3e9c46109b28560eb1d43e42 (table test-table [id=22608006576b4a8995d60716a5df21c3]) successfully deleted
I20260929 15:57:40.032683 7318 catalog_manager.cc:4841] TS 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021): tablet 3e8837de3e9c46109b28560eb1d43e42 (table test-table [id=22608006576b4a8995d60716a5df21c3]) successfully deleted
I20260929 15:57:40.175189 7844 raft_consensus.cc:482] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:40.175565 7844 raft_consensus.cc:504] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:40.178032 7844 leader_election.cc:288] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007)
I20260929 15:57:40.199188 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.180516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:39701 (local address 127.5.117.1:40021)
0929 15:57:40.180774 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:40.180782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:40.191050 (+ 10268us) server_negotiation.cc:407] Connection header received
0929 15:57:40.191358 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:40.191364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:40.191456 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:40.191519 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:40.192480 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.193648 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.196457 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.197734 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.197838 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.197985 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:40.198677 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 15:57:40.198826 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 15:57:40.199285 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.179815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:49787 (local address 127.5.117.2:39007)
0929 15:57:40.180303 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:40.180312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:40.190605 (+ 10293us) server_negotiation.cc:407] Connection header received
0929 15:57:40.191216 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:40.191224 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:40.191322 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:40.191390 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:40.192708 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.193911 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.196603 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.197907 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.198009 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.198148 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:40.198832 (+ 684us) server_negotiation.cc:299] Negotiation successful
0929 15:57:40.198995 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260929 15:57:40.200352 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.179854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40021 (local address 127.5.117.3:39701)
0929 15:57:40.190908 (+ 11054us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:40.190929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:40.191175 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:40.191722 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:40.191726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:40.191749 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:40.192283 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.192301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.193861 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.193866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.196281 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.196296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.198205 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.198211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.198463 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.198498 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:57:40.199652 (+ 1154us) client_negotiation.cc:240] Negotiation successful
0929 15:57:40.199874 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10880,"mutex_wait_us":5,"thread_start_us":10776,"threads_started":1}
I20260929 15:57:40.201174 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8b32a75c9f40b99a543e7b8d162a81" candidate_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" is_pre_election: true
I20260929 15:57:40.201298 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.179265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.5.117.3:49787)
0929 15:57:40.190380 (+ 11115us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:40.190420 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:40.190978 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:40.191675 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:40.191682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:40.191711 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:40.192455 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.192470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.194176 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.194182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.196430 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.196444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.198452 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.198458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.198629 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.198665 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:57:40.199942 (+ 1277us) client_negotiation.cc:240] Negotiation successful
0929 15:57:40.200126 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10910,"mutex_wait_us":47,"thread_start_us":10747,"threads_started":1}
I20260929 15:57:40.201521 7491 raft_consensus.cc:2454] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ddfc1e32b4c8ebf0fba373bbf7138 in term 0.
I20260929 15:57:40.202266 7700 leader_election.cc:302] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:40.202374 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8b32a75c9f40b99a543e7b8d162a81" candidate_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a" is_pre_election: true
I20260929 15:57:40.202685 7630 raft_consensus.cc:2454] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed6ddfc1e32b4c8ebf0fba373bbf7138 in term 0.
I20260929 15:57:40.203020 7844 raft_consensus.cc:2781] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:40.203146 7844 raft_consensus.cc:482] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:40.203246 7844 raft_consensus.cc:3037] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:40.206809 7844 raft_consensus.cc:504] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:40.208127 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8b32a75c9f40b99a543e7b8d162a81" candidate_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a"
I20260929 15:57:40.208204 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8b32a75c9f40b99a543e7b8d162a81" candidate_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
I20260929 15:57:40.208442 7630 raft_consensus.cc:3037] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:40.208498 7491 raft_consensus.cc:3037] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:40.212222 7844 leader_election.cc:288] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [CANDIDATE]: Term 1 election: Requested vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007)
I20260929 15:57:40.212352 7491 raft_consensus.cc:2454] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ddfc1e32b4c8ebf0fba373bbf7138 in term 1.
I20260929 15:57:40.212456 7630 raft_consensus.cc:2454] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed6ddfc1e32b4c8ebf0fba373bbf7138 in term 1.
I20260929 15:57:40.213156 7700 leader_election.cc:302] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:40.213899 7844 raft_consensus.cc:2781] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:40.214495 7844 raft_consensus.cc:686] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 LEADER]: Becoming Leader. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Running, Role: LEADER
I20260929 15:57:40.215034 7844 consensus_queue.cc:237] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:40.220403 7333 catalog_manager.cc:5495] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } health_report { overall_health: HEALTHY } } }
I20260929 15:57:40.304661 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.296254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.5.117.3:34839)
0929 15:57:40.297343 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:40.297352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:40.297381 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:57:40.297721 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:40.297732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:40.297926 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:40.298166 (+ 240us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:40.298791 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.299749 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.301346 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.302083 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.302231 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.302748 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:40.303763 (+ 1015us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:40.303935 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:40.304094 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 15:57:40.304202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":739,"threads_started":1}
I20260929 15:57:40.325351 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.318415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:35147 (local address 127.5.117.2:39007)
0929 15:57:40.318639 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:40.318644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:40.319339 (+ 695us) server_negotiation.cc:407] Connection header received
0929 15:57:40.319855 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:40.319861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:40.319965 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:40.320146 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:40.320814 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.321538 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.322997 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.323402 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.323534 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.324154 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:40.324492 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:40.324642 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:40.324812 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 15:57:40.324913 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:57:40.325608 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.318415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.5.117.3:35147)
0929 15:57:40.319226 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:40.319244 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:40.319422 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:40.320197 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:40.320204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:40.320222 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:40.320624 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.320634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.321683 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.321688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.322849 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.322861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.323604 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.323612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.323757 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.323837 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:40.324640 (+ 803us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:40.324657 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 15:57:40.324772 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 15:57:40.325066 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":493,"threads_started":1}
I20260929 15:57:40.338241 7630 raft_consensus.cc:1261] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Refusing update from remote peer ed6ddfc1e32b4c8ebf0fba373bbf7138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:40.338406 7491 raft_consensus.cc:1261] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Refusing update from remote peer ed6ddfc1e32b4c8ebf0fba373bbf7138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:40.339164 7844 consensus_queue.cc:1035] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:40.339493 7900 consensus_queue.cc:1035] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:40.358877 7868 mvcc.cc:205] Tried to move back new op lower bound from 7334696797539262464 to 7334696797046435840. Current Snapshot: MvccSnapshot[applied={T|T < 7334696797539262464}]
I20260929 15:57:40.374964 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.368287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:44939 (local address 127.5.117.3:34839)
0929 15:57:40.368467 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:40.368474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:40.368836 (+ 362us) server_negotiation.cc:407] Connection header received
0929 15:57:40.369057 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:40.369065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:40.369156 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:40.369255 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:40.369874 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.370858 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.372828 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.373512 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.373631 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.374175 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:40.374307 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:40.374389 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:40.374563 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:57:40.374645 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:57:40.375507 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:40.368539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34839 (local address 127.5.117.2:44939)
0929 15:57:40.368720 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:40.368737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:40.368891 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:40.369460 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:40.369466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:40.369480 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:40.369701 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.369715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.371007 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.371013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.372673 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:40.372689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:40.373687 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:40.373693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:40.373919 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:40.373979 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:40.374421 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:40.374434 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 15:57:40.374534 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 15:57:40.374700 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 15:57:40.377485 7872 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:57:41.988005 7333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:41.988377 7333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:41.994031 7333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Sending DeleteTablet for 3 replicas of tablet 9e8b32a75c9f40b99a543e7b8d162a81
I20260929 15:57:41.995065 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:41 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42568
I20260929 15:57:41.995270 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:41 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54122
I20260929 15:57:41.996110 7610 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8b32a75c9f40b99a543e7b8d162a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:41 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:57472
I20260929 15:57:41.996686 7929 tablet_replica.cc:330] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: stopping tablet replica
I20260929 15:57:41.996914 7929 raft_consensus.cc:2229] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:41.997087 7931 tablet_replica.cc:330] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: stopping tablet replica
I20260929 15:57:41.997226 7929 raft_consensus.cc:2258] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:41.997634 7930 tablet_replica.cc:330] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: stopping tablet replica
I20260929 15:57:41.997856 7931 raft_consensus.cc:2229] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:41.997957 7930 raft_consensus.cc:2229] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:41.998211 7931 raft_consensus.cc:2258] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:41.998492 7929 ts_tablet_manager.cc:1905] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:41.999347 7930 raft_consensus.cc:2258] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:41.999413 7931 ts_tablet_manager.cc:1905] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:42.001088 7930 ts_tablet_manager.cc:1905] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:42.008208 7930 ts_tablet_manager.cc:1918] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:42.008378 7930 log.cc:1196] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/9e8b32a75c9f40b99a543e7b8d162a81
I20260929 15:57:42.008560 7931 ts_tablet_manager.cc:1918] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:42.008905 7931 log.cc:1196] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/9e8b32a75c9f40b99a543e7b8d162a81
I20260929 15:57:42.008961 7930 ts_tablet_manager.cc:1939] T 9e8b32a75c9f40b99a543e7b8d162a81 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting consensus metadata
I20260929 15:57:42.009552 7931 ts_tablet_manager.cc:1939] T 9e8b32a75c9f40b99a543e7b8d162a81 P 660858a14c774bd5a44400bd02ec265a: Deleting consensus metadata
I20260929 15:57:42.010454 7319 catalog_manager.cc:4841] TS ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839): tablet 9e8b32a75c9f40b99a543e7b8d162a81 (table test-table [id=b725b31f2b5e4b76850b3495e838cbcf]) successfully deleted
I20260929 15:57:42.011066 7321 catalog_manager.cc:4841] TS 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007): tablet 9e8b32a75c9f40b99a543e7b8d162a81 (table test-table [id=b725b31f2b5e4b76850b3495e838cbcf]) successfully deleted
I20260929 15:57:42.011675 7929 ts_tablet_manager.cc:1918] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:42.011830 7929 log.cc:1196] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/9e8b32a75c9f40b99a543e7b8d162a81
I20260929 15:57:42.012333 7929 ts_tablet_manager.cc:1939] T 9e8b32a75c9f40b99a543e7b8d162a81 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting consensus metadata
I20260929 15:57:42.013348 7318 catalog_manager.cc:4841] TS 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021): tablet 9e8b32a75c9f40b99a543e7b8d162a81 (table test-table [id=b725b31f2b5e4b76850b3495e838cbcf]) successfully deleted
Sep 29 15:57:42 dist-test-slave-b69v krb5kdc[7289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:42 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:42.066356 7939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.047004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.5.117.62:35293)
0929 15:57:42.047706 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.047713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.047740 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:42.047956 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.047960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.048034 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.048222 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:42.049676 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.050775 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.052037 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.052577 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.052727 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.058460 (+ 5733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:42.058466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:42.058471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:42.058508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:42.062955 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.063810 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.063815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.063820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.063981 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.064517 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.064522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.064525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.064899 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:42.065177 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.065476 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.065759 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":346,"threads_started":1}
I20260929 15:57:42.070009 7333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:42.070669 7333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:42.077473 7471 tablet_service.cc:1467] Processing CreateTablet for tablet c310dfad53bb42e7a00cfa974c69de51 (DEFAULT_TABLE table=test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.077548 7610 tablet_service.cc:1467] Processing CreateTablet for tablet c310dfad53bb42e7a00cfa974c69de51 (DEFAULT_TABLE table=test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.078002 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dfad53bb42e7a00cfa974c69de51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.078045 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dfad53bb42e7a00cfa974c69de51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.077942 7749 tablet_service.cc:1467] Processing CreateTablet for tablet c310dfad53bb42e7a00cfa974c69de51 (DEFAULT_TABLE table=test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.078368 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c310dfad53bb42e7a00cfa974c69de51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.084203 7940 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Bootstrap starting.
I20260929 15:57:42.085958 7940 tablet_bootstrap.cc:654] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.088526 7940 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: No bootstrap required, opened a new log
I20260929 15:57:42.088589 7941 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Bootstrap starting.
I20260929 15:57:42.088675 7940 ts_tablet_manager.cc:1397] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:57:42.089313 7940 raft_consensus.cc:348] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.089532 7940 raft_consensus.cc:374] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.089633 7940 raft_consensus.cc:729] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.089879 7940 consensus_queue.cc:260] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.090209 7941 tablet_bootstrap.cc:654] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.090751 7940 ts_tablet_manager.cc:1428] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:42.091014 7942 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Bootstrap starting.
I20260929 15:57:42.092707 7941 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: No bootstrap required, opened a new log
I20260929 15:57:42.092870 7941 ts_tablet_manager.cc:1397] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 15:57:42.092941 7942 tablet_bootstrap.cc:654] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.093674 7941 raft_consensus.cc:348] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.093941 7941 raft_consensus.cc:374] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.094063 7941 raft_consensus.cc:729] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.094323 7941 consensus_queue.cc:260] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.094874 7941 ts_tablet_manager.cc:1428] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:57:42.096378 7942 tablet_bootstrap.cc:492] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: No bootstrap required, opened a new log
I20260929 15:57:42.096534 7942 ts_tablet_manager.cc:1397] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 15:57:42.097291 7942 raft_consensus.cc:348] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.097576 7942 raft_consensus.cc:374] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.097707 7942 raft_consensus.cc:729] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.098002 7942 consensus_queue.cc:260] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.098642 7942 ts_tablet_manager.cc:1428] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:57:42.414239 7944 raft_consensus.cc:482] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:42.414551 7944 raft_consensus.cc:504] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.415598 7944 leader_election.cc:288] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:42.416074 7629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dfad53bb42e7a00cfa974c69de51" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a" is_pre_election: true
I20260929 15:57:42.416105 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dfad53bb42e7a00cfa974c69de51" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" is_pre_election: true
I20260929 15:57:42.416399 7629 raft_consensus.cc:2454] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 0.
I20260929 15:57:42.416421 7769 raft_consensus.cc:2454] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 0.
I20260929 15:57:42.416914 7422 leader_election.cc:302] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:42.417186 7944 raft_consensus.cc:2781] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:42.417308 7944 raft_consensus.cc:482] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:42.417412 7944 raft_consensus.cc:3037] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.419736 7944 raft_consensus.cc:504] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.420317 7944 leader_election.cc:288] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 election: Requested vote from peers 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:42.420733 7629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dfad53bb42e7a00cfa974c69de51" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "660858a14c774bd5a44400bd02ec265a"
I20260929 15:57:42.420801 7769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c310dfad53bb42e7a00cfa974c69de51" candidate_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
I20260929 15:57:42.420943 7629 raft_consensus.cc:3037] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.421011 7769 raft_consensus.cc:3037] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.424214 7629 raft_consensus.cc:2454] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 1.
I20260929 15:57:42.424206 7769 raft_consensus.cc:2454] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1487d31f6e3f4f09bc71d2c8bb91b5ed in term 1.
I20260929 15:57:42.424639 7424 leader_election.cc:302] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, 660858a14c774bd5a44400bd02ec265a, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:42.424929 7944 raft_consensus.cc:2781] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:42.425395 7944 raft_consensus.cc:686] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 LEADER]: Becoming Leader. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Running, Role: LEADER
I20260929 15:57:42.425747 7944 consensus_queue.cc:237] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.428556 7333 catalog_manager.cc:5495] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1). New cstate: current_term: 1 leader_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:42.463550 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.458013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.5.117.1:40021)
0929 15:57:42.458957 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.458961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.458979 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:57:42.459029 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.459033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.459100 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.459184 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:42.459588 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.460349 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.461738 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.462158 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.462248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.462688 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:42.462804 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:42.462876 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.463062 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.463126 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":700,"threads_started":1}
I20260929 15:57:42.472643 7769 raft_consensus.cc:1261] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Refusing update from remote peer 1487d31f6e3f4f09bc71d2c8bb91b5ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:42.472625 7629 raft_consensus.cc:1261] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Refusing update from remote peer 1487d31f6e3f4f09bc71d2c8bb91b5ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:42.473323 7947 consensus_queue.cc:1035] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:42.473703 7944 consensus_queue.cc:1035] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:42.527935 7333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:42.528281 7333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:42.531255 7333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Sending DeleteTablet for 3 replicas of tablet c310dfad53bb42e7a00cfa974c69de51
I20260929 15:57:42.532027 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet c310dfad53bb42e7a00cfa974c69de51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:42 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42568
I20260929 15:57:42.532109 7610 tablet_service.cc:1514] Processing DeleteTablet for tablet c310dfad53bb42e7a00cfa974c69de51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:42 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:57472
I20260929 15:57:42.532173 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet c310dfad53bb42e7a00cfa974c69de51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:42 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54122
I20260929 15:57:42.533546 7333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:42.534441 7333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:42.536346 7959 tablet_replica.cc:330] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: stopping tablet replica
I20260929 15:57:42.536504 7957 tablet_replica.cc:330] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: stopping tablet replica
I20260929 15:57:42.536585 7959 raft_consensus.cc:2229] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:42.536696 7958 tablet_replica.cc:330] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: stopping tablet replica
I20260929 15:57:42.536693 7957 raft_consensus.cc:2229] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:42.536906 7958 raft_consensus.cc:2229] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:42.536960 7959 raft_consensus.cc:2258] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:42.537132 7958 raft_consensus.cc:2258] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:42.537135 7957 raft_consensus.cc:2258] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:42.538285 7959 ts_tablet_manager.cc:1905] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:42.538398 7958 ts_tablet_manager.cc:1905] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:42.538421 7957 ts_tablet_manager.cc:1905] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:42.545324 7610 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb (DEFAULT_TABLE table=test-table [id=4f7fd920ee0f462eae573d6b4bec466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.545491 7958 ts_tablet_manager.cc:1918] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:42.545640 7958 log.cc:1196] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/c310dfad53bb42e7a00cfa974c69de51
I20260929 15:57:42.545622 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb (DEFAULT_TABLE table=test-table [id=4f7fd920ee0f462eae573d6b4bec466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.545836 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6870a5e4e64a62b9c478c8643694fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.546056 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6870a5e4e64a62b9c478c8643694fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.546257 7958 ts_tablet_manager.cc:1939] T c310dfad53bb42e7a00cfa974c69de51 P 660858a14c774bd5a44400bd02ec265a: Deleting consensus metadata
I20260929 15:57:42.546994 7471 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb (DEFAULT_TABLE table=test-table [id=4f7fd920ee0f462eae573d6b4bec466a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:42.548115 7321 catalog_manager.cc:4841] TS 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007): tablet c310dfad53bb42e7a00cfa974c69de51 (table test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]) successfully deleted
I20260929 15:57:42.549211 7959 ts_tablet_manager.cc:1918] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:42.549367 7959 log.cc:1196] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/c310dfad53bb42e7a00cfa974c69de51
I20260929 15:57:42.549970 7959 ts_tablet_manager.cc:1939] T c310dfad53bb42e7a00cfa974c69de51 P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting consensus metadata
I20260929 15:57:42.551174 7319 catalog_manager.cc:4841] TS ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839): tablet c310dfad53bb42e7a00cfa974c69de51 (table test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]) successfully deleted
I20260929 15:57:42.552773 7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6870a5e4e64a62b9c478c8643694fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:42.552721 7957 ts_tablet_manager.cc:1918] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:42.553030 7942 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Bootstrap starting.
I20260929 15:57:42.553057 7957 log.cc:1196] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/c310dfad53bb42e7a00cfa974c69de51
I20260929 15:57:42.553406 7941 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Bootstrap starting.
I20260929 15:57:42.553815 7957 ts_tablet_manager.cc:1939] T c310dfad53bb42e7a00cfa974c69de51 P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting consensus metadata
I20260929 15:57:42.555382 7942 tablet_bootstrap.cc:654] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.555441 7941 tablet_bootstrap.cc:654] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.555404 7318 catalog_manager.cc:4841] TS 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021): tablet c310dfad53bb42e7a00cfa974c69de51 (table test-table [id=1d935e2a5ef849f2aa0bdfc1a00e27bb]) successfully deleted
I20260929 15:57:42.560741 7940 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Bootstrap starting.
I20260929 15:57:42.562683 7941 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: No bootstrap required, opened a new log
I20260929 15:57:42.562708 7942 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: No bootstrap required, opened a new log
I20260929 15:57:42.563040 7942 ts_tablet_manager.cc:1397] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260929 15:57:42.563182 7940 tablet_bootstrap.cc:654] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:42.563182 7941 ts_tablet_manager.cc:1397] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 15:57:42.564242 7942 raft_consensus.cc:348] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.564432 7941 raft_consensus.cc:348] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.564623 7942 raft_consensus.cc:374] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.564738 7942 raft_consensus.cc:729] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.564749 7941 raft_consensus.cc:374] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.564841 7941 raft_consensus.cc:729] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6ddfc1e32b4c8ebf0fba373bbf7138, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.565158 7941 consensus_queue.cc:260] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.565150 7942 consensus_queue.cc:260] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.565816 7941 ts_tablet_manager.cc:1428] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:57:42.565927 7942 ts_tablet_manager.cc:1428] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:42.568127 7940 tablet_bootstrap.cc:492] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: No bootstrap required, opened a new log
I20260929 15:57:42.568485 7940 ts_tablet_manager.cc:1397] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:57:42.569537 7940 raft_consensus.cc:348] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.569862 7940 raft_consensus.cc:374] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:42.569957 7940 raft_consensus.cc:729] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1487d31f6e3f4f09bc71d2c8bb91b5ed, State: Initialized, Role: FOLLOWER
I20260929 15:57:42.570276 7940 consensus_queue.cc:260] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.570961 7940 ts_tablet_manager.cc:1428] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:57:42.783991 7949 raft_consensus.cc:482] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:42.784356 7949 raft_consensus.cc:504] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.785292 7949 leader_election.cc:288] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:42.786108 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6870a5e4e64a62b9c478c8643694fb" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" is_pre_election: true
I20260929 15:57:42.786116 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6870a5e4e64a62b9c478c8643694fb" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" is_pre_election: true
I20260929 15:57:42.786418 7490 raft_consensus.cc:2454] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 0.
I20260929 15:57:42.786434 7768 raft_consensus.cc:2454] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 0.
I20260929 15:57:42.786964 7562 leader_election.cc:302] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 660858a14c774bd5a44400bd02ec265a, ed6ddfc1e32b4c8ebf0fba373bbf7138; no voters:
I20260929 15:57:42.787274 7949 raft_consensus.cc:2781] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:42.787427 7949 raft_consensus.cc:482] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:42.787561 7949 raft_consensus.cc:3037] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.790163 7949 raft_consensus.cc:504] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.790757 7949 leader_election.cc:288] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 election: Requested vote from peers 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021), ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839)
I20260929 15:57:42.791216 7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6870a5e4e64a62b9c478c8643694fb" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed"
I20260929 15:57:42.791416 7490 raft_consensus.cc:3037] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.791352 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6870a5e4e64a62b9c478c8643694fb" candidate_uuid: "660858a14c774bd5a44400bd02ec265a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138"
I20260929 15:57:42.791589 7768 raft_consensus.cc:3037] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:42.793975 7490 raft_consensus.cc:2454] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 1.
I20260929 15:57:42.794234 7768 raft_consensus.cc:2454] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 660858a14c774bd5a44400bd02ec265a in term 1.
I20260929 15:57:42.794517 7561 leader_election.cc:302] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1487d31f6e3f4f09bc71d2c8bb91b5ed, 660858a14c774bd5a44400bd02ec265a; no voters:
I20260929 15:57:42.794852 7949 raft_consensus.cc:2781] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:42.795037 7949 raft_consensus.cc:686] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 1 LEADER]: Becoming Leader. State: Replica: 660858a14c774bd5a44400bd02ec265a, State: Running, Role: LEADER
I20260929 15:57:42.795372 7949 consensus_queue.cc:237] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } }
I20260929 15:57:42.799341 7333 catalog_manager.cc:5495] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a reported cstate change: term changed from 0 to 1, leader changed from <none> to 660858a14c774bd5a44400bd02ec265a (127.5.117.2). New cstate: current_term: 1 leader_uuid: "660858a14c774bd5a44400bd02ec265a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "660858a14c774bd5a44400bd02ec265a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:42.876951 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.871167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.5.117.2:39007)
0929 15:57:42.871942 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.871946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.871965 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:42.872033 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.872036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.872127 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.872241 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:42.872681 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.873555 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.874982 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.875428 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.875565 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.876032 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:42.876175 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:42.876280 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.876437 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.876516 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":504,"threads_started":1}
I20260929 15:57:42.895310 7964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.889037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35407 (local address 127.5.117.2:39007)
0929 15:57:42.889249 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.889255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.890041 (+ 786us) server_negotiation.cc:407] Connection header received
0929 15:57:42.890205 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.890209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.890277 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.890360 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:42.891081 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.891897 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.893565 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.893992 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.894096 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.894535 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:42.894667 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:42.894743 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.894907 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.895011 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":81}
I20260929 15:57:42.895704 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.888758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39007 (local address 127.5.117.2:35407)
0929 15:57:42.889920 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:42.889940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:42.890086 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:42.890465 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:42.890471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:42.890486 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:42.890895 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.890906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.892121 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.892125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.893327 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.893339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.894173 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.894178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.894315 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.894354 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:42.894779 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:42.894789 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:57:42.894870 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 15:57:42.895017 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":5,"thread_start_us":629,"threads_started":1}
I20260929 15:57:42.904697 7768 raft_consensus.cc:1261] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Refusing update from remote peer 660858a14c774bd5a44400bd02ec265a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:42.904680 7490 raft_consensus.cc:1261] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Refusing update from remote peer 660858a14c774bd5a44400bd02ec265a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:42.905269 7911 consensus_queue.cc:1035] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed6ddfc1e32b4c8ebf0fba373bbf7138" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:42.905606 7949 consensus_queue.cc:1035] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1487d31f6e3f4f09bc71d2c8bb91b5ed" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:57:42.928357 7909 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:57:44.521555 7334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:44.521893 7334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:44.526957 7334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0669a728c12c4f56a20f3301fe48eaab: Sending DeleteTablet for 3 replicas of tablet 4e6870a5e4e64a62b9c478c8643694fb
I20260929 15:57:44.527721 7471 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42568
I20260929 15:57:44.527882 7609 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:57472
I20260929 15:57:44.528007 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6870a5e4e64a62b9c478c8643694fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54122
I20260929 15:57:44.528731 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7405
I20260929 15:57:44.538352 7992 tablet_replica.cc:330] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: stopping tablet replica
I20260929 15:57:44.538619 7992 raft_consensus.cc:2229] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:44.538978 7992 raft_consensus.cc:2258] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:44.540202 7992 ts_tablet_manager.cc:1905] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:44.540311 7991 tablet_replica.cc:330] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: stopping tablet replica
I20260929 15:57:44.540553 7991 raft_consensus.cc:2229] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:44.541049 7990 tablet_replica.cc:330] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: stopping tablet replica
I20260929 15:57:44.541214 7991 raft_consensus.cc:2258] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:44.541369 7990 raft_consensus.cc:2229] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:44.541734 7990 raft_consensus.cc:2258] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:44.543165 7990 ts_tablet_manager.cc:1905] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:44.543148 7991 ts_tablet_manager.cc:1905] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:44.546955 7992 ts_tablet_manager.cc:1918] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:44.547119 7992 log.cc:1196] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/4e6870a5e4e64a62b9c478c8643694fb
I20260929 15:57:44.547569 7992 ts_tablet_manager.cc:1939] T 4e6870a5e4e64a62b9c478c8643694fb P ed6ddfc1e32b4c8ebf0fba373bbf7138: Deleting consensus metadata
I20260929 15:57:44.548774 7319 catalog_manager.cc:4841] TS ed6ddfc1e32b4c8ebf0fba373bbf7138 (127.5.117.3:34839): tablet 4e6870a5e4e64a62b9c478c8643694fb (table test-table [id=4f7fd920ee0f462eae573d6b4bec466a]) successfully deleted
I20260929 15:57:44.551630 7990 ts_tablet_manager.cc:1918] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:44.551815 7990 log.cc:1196] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/4e6870a5e4e64a62b9c478c8643694fb
I20260929 15:57:44.551841 7991 ts_tablet_manager.cc:1918] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:44.551983 7991 log.cc:1196] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/4e6870a5e4e64a62b9c478c8643694fb
I20260929 15:57:44.552376 7990 ts_tablet_manager.cc:1939] T 4e6870a5e4e64a62b9c478c8643694fb P 1487d31f6e3f4f09bc71d2c8bb91b5ed: Deleting consensus metadata
I20260929 15:57:44.552520 7991 ts_tablet_manager.cc:1939] T 4e6870a5e4e64a62b9c478c8643694fb P 660858a14c774bd5a44400bd02ec265a: Deleting consensus metadata
I20260929 15:57:44.553920 7318 catalog_manager.cc:4841] TS 1487d31f6e3f4f09bc71d2c8bb91b5ed (127.5.117.1:40021): tablet 4e6870a5e4e64a62b9c478c8643694fb (table test-table [id=4f7fd920ee0f462eae573d6b4bec466a]) successfully deleted
I20260929 15:57:44.554266 7321 catalog_manager.cc:4841] TS 660858a14c774bd5a44400bd02ec265a (127.5.117.2:39007): tablet 4e6870a5e4e64a62b9c478c8643694fb (table test-table [id=4f7fd920ee0f462eae573d6b4bec466a]) successfully deleted
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:44.564851 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:44.530622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.5.117.1:40021)
0929 15:57:44.532659 (+ 2037us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:44.532670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:44.533473 (+ 803us) server_negotiation.cc:407] Connection header received
0929 15:57:44.533554 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:44.533563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:44.533663 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:44.533791 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:44.536282 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.537716 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.540414 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.541322 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.541457 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:44.557587 (+ 16130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:44.557607 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:44.557632 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:44.557676 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:44.561538 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.562347 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.562356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.562358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.562662 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.563095 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.563099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.563101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.563493 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:44.563839 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:44.564196 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:57:44.564460 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1842,"thread_start_us":1653,"threads_started":1}
W20260929 15:57:44.565544 7536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45980
W20260929 15:57:44.566224 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7405: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:44.566486 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 7405
I20260929 15:57:44.593235 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7545
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:44.611490 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:44.594073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.5.117.2:39007)
0929 15:57:44.594892 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:44.594897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:44.594925 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:57:44.594990 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:44.594994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:44.595081 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:44.595218 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:44.596640 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.597465 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.599617 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.600257 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.600386 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:44.605387 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:44.605400 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:44.605419 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:44.605457 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:44.608820 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.609579 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.609606 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.609610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.609771 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.610154 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.610157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.610163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.610514 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:44.610798 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:44.611026 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 15:57:44.611199 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":449,"threads_started":1}
W20260929 15:57:44.612300 7675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58856
W20260929 15:57:44.612957 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:44.613113 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 7545
I20260929 15:57:44.641501 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7684
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[7289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:44.658162 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:44.642341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.5.117.3:34839)
0929 15:57:44.643044 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:44.643049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:44.643068 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:44.643131 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:44.643135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:44.643325 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:44.643481 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:44.644903 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.645781 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.647013 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:44.647424 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:44.647523 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:44.652223 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:44.652236 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:44.652256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:44.652294 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:44.655095 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.655695 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.655702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.655704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.655867 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:44.656588 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:44.656592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:44.656594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:44.656990 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:44.657230 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:44.657505 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 15:57:44.657846 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":412,"threads_started":1}
W20260929 15:57:44.658941 7814 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33008
W20260929 15:57:44.659667 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:44.659864 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 7684
I20260929 15:57:44.684631 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 7302
I20260929 15:57:44.686200 7374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:44.819293 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 7302
2026-09-29T15:57:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9463 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[8000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[8000](info): set up 2 sockets
Sep 29 15:57:44 dist-test-slave-b69v krb5kdc[8000](info): commencing operation
krb5kdc: starting...
W20260929 15:57:45.864745 5588 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:46 dist-test-slave-b69v krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:46.075927 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45031
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:46.460572 8013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:46.460925 8013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:46.461125 8013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:46.468941 8013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:46.469069 8013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:46.469183 8013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:46.469293 8013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:46.469377 8013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:46.479965 8013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:45031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:46.481622 8013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:46.483160 8013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:46.496065 8019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.496140 8018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.496150 8021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:46.497076 8013 server_base.cc:1034] running on GCE node
I20260929 15:57:46.497871 8013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:46.499338 8013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:46.500743 8013 hybrid_clock.cc:648] HybridClock initialized: now 1790697466500653 us; error 47 us; skew 500 ppm
I20260929 15:57:46.501313 8013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:46 dist-test-slave-b69v krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:46.504756 8013 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:46.506414 8013 webserver.cc:460] Webserver started at http://127.5.117.62:41389/ using document root <none> and password file <none>
I20260929 15:57:46.507158 8013 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:46.507308 8013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:46.507653 8013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:46.510864 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "e581f7d3f9aa4777884775c1d7adfb88"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-b69v"
server_key: "76a1363cc1fb97d3bab95de826a33339"
server_key_iv: "3838cd87d23dec0c647b2d193a632d54"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.511700 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "e581f7d3f9aa4777884775c1d7adfb88"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-b69v"
server_key: "76a1363cc1fb97d3bab95de826a33339"
server_key_iv: "3838cd87d23dec0c647b2d193a632d54"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.517251 8013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:57:46.520864 8028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.522240 8013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:46.522446 8013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "e581f7d3f9aa4777884775c1d7adfb88"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-b69v"
server_key: "76a1363cc1fb97d3bab95de826a33339"
server_key_iv: "3838cd87d23dec0c647b2d193a632d54"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.522677 8013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:46.542943 8013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:46.547979 8013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:46.548329 8013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:46.569821 8013 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:45031
I20260929 15:57:46.569864 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:45031 every 8 connection(s)
I20260929 15:57:46.571412 8013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:46.576040 8080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:46.577203 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8013
I20260929 15:57:46.577481 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:46.578429 5588 external_mini_cluster.cc:1428] Setting key 5c8b1c16ebd1bdf9909377c20c891913
I20260929 15:57:46.591722 8080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88: Bootstrap starting.
Sep 29 15:57:46 dist-test-slave-b69v krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:46.595685 8080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:46.597092 8080 log.cc:824] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:46.600577 8080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88: No bootstrap required, opened a new log
I20260929 15:57:46.603835 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:46.581449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.5.117.62:45031)
0929 15:57:46.582382 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:46.582396 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:46.582446 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:57:46.584539 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:46.584571 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:46.585111 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:46.585624 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:46.587332 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.588845 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.590189 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.590912 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.591145 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:46.595766 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:46.595781 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:46.595804 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:46.595863 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:46.599530 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.600318 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.600326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.600332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.600498 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.600934 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.600938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.600940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.601492 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:46.601838 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:46.602278 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 15:57:46.602747 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":391,"threads_started":1}
I20260929 15:57:46.609584 8080 raft_consensus.cc:348] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } }
I20260929 15:57:46.610023 8080 raft_consensus.cc:374] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:46.610143 8080 raft_consensus.cc:729] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e581f7d3f9aa4777884775c1d7adfb88, State: Initialized, Role: FOLLOWER
I20260929 15:57:46.610781 8080 consensus_queue.cc:260] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } }
I20260929 15:57:46.611049 8080 raft_consensus.cc:388] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:46.611195 8080 raft_consensus.cc:482] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:46.611388 8080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:46.614086 8080 raft_consensus.cc:504] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } }
I20260929 15:57:46.614643 8080 leader_election.cc:302] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e581f7d3f9aa4777884775c1d7adfb88; no voters:
I20260929 15:57:46.615317 8080 leader_election.cc:288] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:46.615458 8085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:46.616240 8085 raft_consensus.cc:686] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [term 1 LEADER]: Becoming Leader. State: Replica: e581f7d3f9aa4777884775c1d7adfb88, State: Running, Role: LEADER
I20260929 15:57:46.616886 8085 consensus_queue.cc:237] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } }
I20260929 15:57:46.617718 8080 sys_catalog.cc:564] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:46.620107 8087 sys_catalog.cc:455] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e581f7d3f9aa4777884775c1d7adfb88. Latest consensus state: current_term: 1 leader_uuid: "e581f7d3f9aa4777884775c1d7adfb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } } }
I20260929 15:57:46.620118 8086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e581f7d3f9aa4777884775c1d7adfb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e581f7d3f9aa4777884775c1d7adfb88" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45031 } } }
I20260929 15:57:46.620571 8086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:46.620571 8087 sys_catalog.cc:458] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:46.622300 8091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:46.627103 8091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:46.636650 8091 catalog_manager.cc:1347] Generated new cluster ID: 05e1263be03b4dce887a9cd8e8aef348
I20260929 15:57:46.636763 8091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:46.656224 8091 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:46.658354 8091 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:46.668414 8091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e581f7d3f9aa4777884775c1d7adfb88: Generated new TSK 0
I20260929 15:57:46.669546 8091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:46.730881 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:47.118422 8106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:47.118949 8106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:47.119226 8106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:47.128891 8106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:47.129063 8106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:47.129258 8106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:47.139657 8106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:47.141503 8106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:47.143121 8106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:47.158438 8111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:47.158515 8112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:47.158473 8114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:47.164299 8106 server_base.cc:1034] running on GCE node
I20260929 15:57:47.164989 8106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:47.166105 8106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:47.167384 8106 hybrid_clock.cc:648] HybridClock initialized: now 1790697467167320 us; error 48 us; skew 500 ppm
I20260929 15:57:47.167793 8106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:47 dist-test-slave-b69v krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:47.172158 8106 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:47.180757 8106 webserver.cc:460] Webserver started at http://127.5.117.1:35689/ using document root <none> and password file <none>
I20260929 15:57:47.182226 8106 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:47.182669 8106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:47.183176 8106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:47.187297 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "f08d2059c37f47cd8bfbab39fa6c4d2f"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-b69v"
server_key: "fb188053334511437f1fa40edec40214"
server_key_iv: "11bdeec9d9eb4f914efe429baee9183f"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.188409 8106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "f08d2059c37f47cd8bfbab39fa6c4d2f"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-b69v"
server_key: "fb188053334511437f1fa40edec40214"
server_key_iv: "11bdeec9d9eb4f914efe429baee9183f"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.196617 8106 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 15:57:47.202850 8121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:47.205276 8106 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 15:57:47.205556 8106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "f08d2059c37f47cd8bfbab39fa6c4d2f"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-b69v"
server_key: "fb188053334511437f1fa40edec40214"
server_key_iv: "11bdeec9d9eb4f914efe429baee9183f"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.205871 8106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:47.220785 8106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:47.227641 8106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:47.228196 8106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:47.229566 8106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:47.231833 8106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:47.231959 8106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:47.232173 8106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:47.232247 8106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:47.274420 8106 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:44347
I20260929 15:57:47.274441 8233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:44347 every 8 connection(s)
I20260929 15:57:47.275947 8106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:47.278540 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8106
I20260929 15:57:47.278831 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:47.279716 5588 external_mini_cluster.cc:1428] Setting key d132aa79196f3b6955358e24f4ee283e
Sep 29 15:57:47 dist-test-slave-b69v krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:47.301352 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.278968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:57711 (local address 127.5.117.62:45031)
0929 15:57:47.279555 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.279563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.280770 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 15:57:47.282817 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.282826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.282923 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.283058 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.284841 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.286092 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.287778 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.288541 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.288699 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.292505 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.292515 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.292524 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.292579 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.296529 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.297769 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.297778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.297784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.298009 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.298566 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.298571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.298575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.299140 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.299410 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.300646 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.300956 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":280,"threads_started":1}
I20260929 15:57:47.302452 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.279563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:45031 (local address 127.5.117.1:57711)
0929 15:57:47.280576 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:47.280636 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:47.282521 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:47.283233 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:47.283241 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:47.283918 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:47.284652 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:47.284670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.286284 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.286288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:47.287619 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:47.287636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.288772 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.288778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:47.288986 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.289785 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:47.289810 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:47.292281 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:47.296865 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:47.296881 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:47.296900 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:47.297576 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:47.298187 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:47.298191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:47.298194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:47.298407 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:47.299557 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:47.299568 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:47.300171 (+ 603us) client_negotiation.cc:769] Sending connection context
0929 15:57:47.300499 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 15:57:47.301139 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":462,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 15:57:47.304737 8234 heartbeater.cc:347] Connected to a master server at 127.5.117.62:45031
I20260929 15:57:47.305073 8234 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:47.306025 8234 heartbeater.cc:511] Master 127.5.117.62:45031 requested a full tablet report, sending...
I20260929 15:57:47.308947 8045 ts_manager.cc:194] Registered new tserver with Master: f08d2059c37f47cd8bfbab39fa6c4d2f (127.5.117.1:44347)
I20260929 15:57:47.311022 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:57711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
I20260929 15:57:48.314998 8234 heartbeater.cc:503] Master 127.5.117.62:45031 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
W20260929 15:57:48.422356 5588 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.117.2: real 1.140s user 0.003s sys 0.012s
I20260929 15:57:48.423910 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:48.798846 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:48.799299 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:48.799499 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:48.807695 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:48.807890 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:48.808127 8240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:48.819376 8240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:48.821091 8240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:48.822661 8240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:48.838276 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:48.838372 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:48.838322 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:48.840377 8240 server_base.cc:1034] running on GCE node
I20260929 15:57:48.841370 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:48.842895 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:48.844197 8240 hybrid_clock.cc:648] HybridClock initialized: now 1790697468844152 us; error 66 us; skew 500 ppm
I20260929 15:57:48.844596 8240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:48 dist-test-slave-b69v krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:48.848325 8240 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:48.850139 8240 webserver.cc:460] Webserver started at http://127.5.117.2:46357/ using document root <none> and password file <none>
I20260929 15:57:48.851037 8240 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:48.851200 8240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:48.851593 8240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:48.855268 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "4116e41396474a54ad39bf34cbf19212"
format_stamp: "Formatted at 2026-09-29 15:57:48 on dist-test-slave-b69v"
server_key: "f35829f2d1cb1abc8c0b797ef8d34b22"
server_key_iv: "49c65faf626418d0c13707c3fdc088c0"
server_key_version: "encryptionkey@0"
I20260929 15:57:48.856259 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "4116e41396474a54ad39bf34cbf19212"
format_stamp: "Formatted at 2026-09-29 15:57:48 on dist-test-slave-b69v"
server_key: "f35829f2d1cb1abc8c0b797ef8d34b22"
server_key_iv: "49c65faf626418d0c13707c3fdc088c0"
server_key_version: "encryptionkey@0"
I20260929 15:57:48.862617 8240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:48.866879 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:48.868988 8240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 15:57:48.869190 8240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "4116e41396474a54ad39bf34cbf19212"
format_stamp: "Formatted at 2026-09-29 15:57:48 on dist-test-slave-b69v"
server_key: "f35829f2d1cb1abc8c0b797ef8d34b22"
server_key_iv: "49c65faf626418d0c13707c3fdc088c0"
server_key_version: "encryptionkey@0"
I20260929 15:57:48.869422 8240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:48.891825 8240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:48.897094 8240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:48.897425 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:48.898566 8240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:48.900354 8240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:48.900467 8240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:48.900642 8240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:48.900722 8240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:48.942624 8240 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44013
I20260929 15:57:48.942657 8367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44013 every 8 connection(s)
I20260929 15:57:48.944547 8240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:48.948529 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8240
I20260929 15:57:48.948757 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:48.949445 5588 external_mini_cluster.cc:1428] Setting key d97203d8fbe13096a6215354d2f96108
Sep 29 15:57:48 dist-test-slave-b69v krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697468, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:48.970796 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:48.947712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:43537 (local address 127.5.117.62:45031)
0929 15:57:48.948299 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:48.948304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:48.949484 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 15:57:48.951771 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:48.951778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:48.951868 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:48.951989 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:48.953810 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:48.955116 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:48.957029 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:48.957767 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:48.957913 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:48.962651 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:48.962661 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:48.962669 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:48.962716 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:48.967005 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:48.967970 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:48.967978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:48.967984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:48.968185 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:48.968621 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:48.968630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:48.968636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:48.969071 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:48.969252 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:48.969955 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 15:57:48.970255 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":345,"threads_started":1}
I20260929 15:57:48.971527 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:48.948331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:45031 (local address 127.5.117.2:43537)
0929 15:57:48.949297 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:48.949347 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:48.951404 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:48.952327 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:48.952337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:48.952958 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:48.953612 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:48.953632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:48.955313 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:48.955317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:48.956845 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:48.956861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:48.958226 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:48.958235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:48.958569 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:48.959464 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:48.959496 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:48.962344 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:48.967245 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:48.967256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:48.967278 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:48.967820 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:48.968303 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:48.968306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:48.968309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:48.968498 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:48.969205 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:48.969211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:48.969556 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 15:57:48.969803 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 15:57:48.970243 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":435,"threads_started":1}
I20260929 15:57:48.973327 8368 heartbeater.cc:347] Connected to a master server at 127.5.117.62:45031
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:57:48.973743 8368 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:48.974697 8368 heartbeater.cc:511] Master 127.5.117.62:45031 requested a full tablet report, sending...
I20260929 15:57:48.976387 8045 ts_manager.cc:194] Registered new tserver with Master: 4116e41396474a54ad39bf34cbf19212 (127.5.117.2:44013)
I20260929 15:57:48.977731 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:43537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
W20260929 15:57:49.091226 5588 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.117.3: real 0.139s user 0.004s sys 0.009s
I20260929 15:57:49.092177 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:45501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:49.449646 8374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:49.449911 8374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:49.450052 8374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:49.456622 8374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:49.456734 8374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:49.456907 8374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:49.466152 8374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:45501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:45031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:49.467468 8374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:49.468988 8374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:49.482527 8379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:49.482453 8382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:49.482492 8380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:49.483328 8374 server_base.cc:1034] running on GCE node
I20260929 15:57:49.483824 8374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:49.484750 8374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:49.485970 8374 hybrid_clock.cc:648] HybridClock initialized: now 1790697469485937 us; error 46 us; skew 500 ppm
I20260929 15:57:49.486265 8374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:49 dist-test-slave-b69v krb5kdc[8000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:49.489409 8374 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:49.490809 8374 webserver.cc:460] Webserver started at http://127.5.117.3:44893/ using document root <none> and password file <none>
I20260929 15:57:49.491490 8374 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:49.491626 8374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:49.491937 8374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:49.495009 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "4554ccfb39274abeb82efa691712d33a"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-b69v"
server_key: "96fcbf383986d17fbb1822e9fc7193fd"
server_key_iv: "fa463835c16933df16b24b35c8d3527c"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.495837 8374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "4554ccfb39274abeb82efa691712d33a"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-b69v"
server_key: "96fcbf383986d17fbb1822e9fc7193fd"
server_key_iv: "fa463835c16933df16b24b35c8d3527c"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.500864 8374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 15:57:49.504307 8389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:49.505604 8374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:57:49.505774 8374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "4554ccfb39274abeb82efa691712d33a"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-b69v"
server_key: "96fcbf383986d17fbb1822e9fc7193fd"
server_key_iv: "fa463835c16933df16b24b35c8d3527c"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.505955 8374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:49.516566 8374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:49.520658 8374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:49.520937 8374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:49.521848 8374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:49.523352 8374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:49.523468 8374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:49.523623 8374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:49.523703 8374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:49.563406 8374 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:33315
I20260929 15:57:49.563424 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:33315 every 8 connection(s)
I20260929 15:57:49.565148 8374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:49.573612 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8374
I20260929 15:57:49.573851 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:49.574416 5588 external_mini_cluster.cc:1428] Setting key bcd6951213acfb55913208c3d65bb9d7
Sep 29 15:57:49 dist-test-slave-b69v krb5kdc[8000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:49.595968 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:49.568206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:52659 (local address 127.5.117.62:45031)
0929 15:57:49.568801 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:49.568806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:49.569866 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 15:57:49.571901 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:49.571907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:49.572009 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:49.572167 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:49.574166 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.575780 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.578138 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.578732 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.578841 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:49.588648 (+ 9807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:49.588657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:49.588661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:49.588709 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:49.592504 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:49.593345 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:49.593355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:49.593359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:49.593486 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:49.593974 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:49.593984 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:49.593986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:49.594297 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:49.594471 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:49.595291 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 15:57:49.595528 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":244,"threads_started":1}
I20260929 15:57:49.597236 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:49.568844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:45031 (local address 127.5.117.3:52659)
0929 15:57:49.569689 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:49.569738 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:49.571608 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:49.572431 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:49.572440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:49.573034 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:49.573933 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:49.573985 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.575974 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.575980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:49.577968 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:49.577984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.578939 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.578944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:49.579155 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:49.585515 (+ 6360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:49.585546 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:49.588344 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:49.592660 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:49.592670 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:49.592693 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:49.593208 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:49.593618 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:49.593623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:49.593627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:49.593840 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:49.594446 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:49.594455 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:49.594875 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 15:57:49.595187 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 15:57:49.595852 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":316,"threads_started":1}
I20260929 15:57:49.598909 8502 heartbeater.cc:347] Connected to a master server at 127.5.117.62:45031
I20260929 15:57:49.599292 8502 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:49.600224 8502 heartbeater.cc:511] Master 127.5.117.62:45031 requested a full tablet report, sending...
I20260929 15:57:49.602486 8045 ts_manager.cc:194] Registered new tserver with Master: 4554ccfb39274abeb82efa691712d33a (127.5.117.3:33315)
I20260929 15:57:49.603615 8045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:52659
I20260929 15:57:49.603806 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:57:49.640167 8504 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:49.637325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.5.117.62:45031)
0929 15:57:49.637540 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:49.637545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:49.638040 (+ 495us) server_negotiation.cc:407] Connection header received
0929 15:57:49.638288 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:49.638293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:49.638376 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:49.638494 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:49.639887 (+ 1393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39376: BlockingRecv error: recv got EOF from 127.0.0.1:39376 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:49.643606 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8106
W20260929 15:57:49.646531 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8106: Not authorized: Client connection negotiation failed: client connection to 127.5.117.1:44347: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:49.646744 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8106
I20260929 15:57:49.664880 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8240
W20260929 15:57:49.668115 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8240: Not authorized: Client connection negotiation failed: client connection to 127.5.117.2:44013: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:49.668300 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8240
I20260929 15:57:49.686060 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8374
W20260929 15:57:49.689862 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8374: Not authorized: Client connection negotiation failed: client connection to 127.5.117.3:33315: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:49.689996 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8374
I20260929 15:57:49.706877 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8013
I20260929 15:57:49.708294 8075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:49.824947 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8013
2026-09-29T15:57:49Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5004 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:49 dist-test-slave-b69v krb5kdc[8520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:49 dist-test-slave-b69v krb5kdc[8520](info): set up 2 sockets
Sep 29 15:57:49 dist-test-slave-b69v krb5kdc[8520](info): commencing operation
krb5kdc: starting...
W20260929 15:57:50.908803 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:51.119975 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39939
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:51.504231 8533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:51.504554 8533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:51.504710 8533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:51.511955 8533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:51.512111 8533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:51.512223 8533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:51.512339 8533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:51.512434 8533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:51.522485 8533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:51.523865 8533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:51.525333 8533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:51.538195 8538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:51.538139 8539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:51.538234 8541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:51.539294 8533 server_base.cc:1034] running on GCE node
I20260929 15:57:51.540211 8533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:51.541636 8533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:51.542840 8533 hybrid_clock.cc:648] HybridClock initialized: now 1790697471542800 us; error 27 us; skew 500 ppm
I20260929 15:57:51.543190 8533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697471, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:51.546427 8533 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:51.547856 8533 webserver.cc:460] Webserver started at http://127.5.117.62:41591/ using document root <none> and password file <none>
I20260929 15:57:51.548583 8533 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:51.548723 8533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:51.549058 8533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:51.552256 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "ca3288f0c77b4541a25178ebc29ac3b8"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-b69v"
server_key: "88f09174e71b5831783e4c66f2f90181"
server_key_iv: "c60aa12a38d9d64028e26b6d35566d57"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.553071 8533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "ca3288f0c77b4541a25178ebc29ac3b8"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-b69v"
server_key: "88f09174e71b5831783e4c66f2f90181"
server_key_iv: "c60aa12a38d9d64028e26b6d35566d57"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.558077 8533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:57:51.561479 8548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:51.562692 8533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:51.562875 8533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "ca3288f0c77b4541a25178ebc29ac3b8"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-b69v"
server_key: "88f09174e71b5831783e4c66f2f90181"
server_key_iv: "c60aa12a38d9d64028e26b6d35566d57"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.563058 8533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:51.578619 8533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:51.582192 8533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:51.582445 8533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:51.602540 8533 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39939
I20260929 15:57:51.602561 8599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39939 every 8 connection(s)
I20260929 15:57:51.604171 8533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:51.608592 8600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:51.612373 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8533
I20260929 15:57:51.612597 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:51.613318 5588 external_mini_cluster.cc:1428] Setting key a2dabb5ecd31721b5214664cd8d32bab
I20260929 15:57:51.623042 8600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8: Bootstrap starting.
I20260929 15:57:51.626974 8600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:51.628154 8600 log.cc:824] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:51.631704 8600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8: No bootstrap required, opened a new log
I20260929 15:57:51.639324 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.615432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.5.117.62:39939)
0929 15:57:51.616334 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.616351 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.616406 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:57:51.618551 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.618587 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.619171 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.619633 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.621109 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.623160 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.624381 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.625547 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.625742 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.630192 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.630204 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.630223 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.630272 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.634482 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.635194 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.635208 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.635218 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.635452 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.635815 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.635820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.635824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.636637 (+ 813us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.636972 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.637266 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.637767 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":343,"threads_started":1}
I20260929 15:57:51.643019 8600 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } }
I20260929 15:57:51.643487 8600 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:51.643631 8600 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3288f0c77b4541a25178ebc29ac3b8, State: Initialized, Role: FOLLOWER
I20260929 15:57:51.643838 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:51.644501 8600 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } }
I20260929 15:57:51.644795 8600 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:51.644960 8600 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:51.645138 8600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:51.649092 8600 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } }
I20260929 15:57:51.649784 8600 leader_election.cc:302] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3288f0c77b4541a25178ebc29ac3b8; no voters:
I20260929 15:57:51.650494 8600 leader_election.cc:288] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:51.650624 8605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:51.651976 8605 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [term 1 LEADER]: Becoming Leader. State: Replica: ca3288f0c77b4541a25178ebc29ac3b8, State: Running, Role: LEADER
I20260929 15:57:51.652608 8605 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } }
I20260929 15:57:51.653151 8600 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:51.655535 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3288f0c77b4541a25178ebc29ac3b8. Latest consensus state: current_term: 1 leader_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } } }
I20260929 15:57:51.655944 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:51.656510 8606 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3288f0c77b4541a25178ebc29ac3b8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39939 } } }
I20260929 15:57:51.656864 8606 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:51.679220 8611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:51.692377 8611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:51.711472 8611 catalog_manager.cc:1347] Generated new cluster ID: a9e757279aff4320ba5af33587a2484c
I20260929 15:57:51.711663 8611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:51.741868 8611 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:51.742767 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.724696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.5.117.62:39939)
0929 15:57:51.724976 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.724988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.732260 (+ 7272us) server_negotiation.cc:407] Connection header received
0929 15:57:51.732892 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.732899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.733028 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.733109 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.734683 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.735877 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.737816 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.738458 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.738562 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.740253 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.740259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.740264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:51.741420 (+ 1156us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.741513 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:51.741541 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.741626 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.742303 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.742425 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:57:51.744729 8611 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:51.755792 8611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca3288f0c77b4541a25178ebc29ac3b8: Generated new TSK 0
I20260929 15:57:51.756898 8611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:51.764531 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8533
I20260929 15:57:51.765851 8595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:51.861749 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8533
2026-09-29T15:57:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2032 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8635](info): set up 2 sockets
Sep 29 15:57:51 dist-test-slave-b69v krb5kdc[8635](info): commencing operation
krb5kdc: starting...
W20260929 15:57:52.969292 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:53 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:53.212118 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41845
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:41845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:53.588443 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:53.588800 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:53.588972 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:53.596875 8648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:53.597002 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:53.597054 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:53.597172 8648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:53.597275 8648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:53.608630 8648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:41845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:53.610459 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:53.612083 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:53.626066 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.626082 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.626135 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:53.629503 8648 server_base.cc:1034] running on GCE node
I20260929 15:57:53.630460 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:53.632311 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:53.633518 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790697473633482 us; error 40 us; skew 500 ppm
I20260929 15:57:53.633914 8648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:53 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697473, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:53.637421 8648 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:53.639297 8648 webserver.cc:460] Webserver started at http://127.5.117.62:45841/ using document root <none> and password file <none>
I20260929 15:57:53.640203 8648 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:53.640357 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:53.640720 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:53.644168 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "f0be1447bee047408525825c73a67600"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-b69v"
server_key: "d6fce96fce9daac7c0c543ebbd77cd3c"
server_key_iv: "64a517d7943ba45bc5bc1c6b7bfb8ce2"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.645026 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "f0be1447bee047408525825c73a67600"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-b69v"
server_key: "d6fce96fce9daac7c0c543ebbd77cd3c"
server_key_iv: "64a517d7943ba45bc5bc1c6b7bfb8ce2"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.650787 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:53.654484 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.655923 8648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:57:53.656169 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "f0be1447bee047408525825c73a67600"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-b69v"
server_key: "d6fce96fce9daac7c0c543ebbd77cd3c"
server_key_iv: "64a517d7943ba45bc5bc1c6b7bfb8ce2"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.656375 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:53.685433 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:53.690992 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:53.691362 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:53.712378 8648 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:41845
I20260929 15:57:53.712401 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:41845 every 8 connection(s)
I20260929 15:57:53.714042 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:53.717871 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8648
I20260929 15:57:53.718144 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:53.719085 5588 external_mini_cluster.cc:1428] Setting key fcd6c345e4b780edeaef69c1975de716
I20260929 15:57:53.719523 8715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:57:53 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:53.735474 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600: Bootstrap starting.
I20260929 15:57:53.739032 8715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:53.740278 8715 log.cc:824] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:53.743976 8715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600: No bootstrap required, opened a new log
I20260929 15:57:53.744199 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:53.721509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.5.117.62:41845)
0929 15:57:53.722381 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:53.722395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:53.722443 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:57:53.724739 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:53.724763 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:53.725196 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:53.725584 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:53.727421 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:53.728883 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:53.730698 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:53.731431 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:53.731582 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:53.736707 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:53.736715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:53.736730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:53.736765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:53.739938 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:53.740714 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:53.740721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:53.740727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:53.740900 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:53.741298 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:53.741303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:53.741311 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:53.741875 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:53.742230 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:53.742675 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 15:57:53.743102 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":396,"threads_started":1}
I20260929 15:57:53.752267 8715 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } }
I20260929 15:57:53.752641 8715 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:53.752763 8715 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0be1447bee047408525825c73a67600, State: Initialized, Role: FOLLOWER
I20260929 15:57:53.753453 8715 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } }
I20260929 15:57:53.753747 8715 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:53.753856 8715 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:53.753989 8715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:53.756292 8715 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } }
I20260929 15:57:53.756810 8715 leader_election.cc:302] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0be1447bee047408525825c73a67600; no voters:
I20260929 15:57:53.757385 8715 leader_election.cc:288] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:53.757861 8720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:53.759078 8720 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [term 1 LEADER]: Becoming Leader. State: Replica: f0be1447bee047408525825c73a67600, State: Running, Role: LEADER
I20260929 15:57:53.759908 8715 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:53.759838 8720 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } }
I20260929 15:57:53.763821 8721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0be1447bee047408525825c73a67600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } } }
I20260929 15:57:53.763790 8722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0be1447bee047408525825c73a67600. Latest consensus state: current_term: 1 leader_uuid: "f0be1447bee047408525825c73a67600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0be1447bee047408525825c73a67600" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41845 } } }
I20260929 15:57:53.764398 8722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:53.764396 8721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:53.766135 8729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:53.772104 8729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:53.782316 8729 catalog_manager.cc:1347] Generated new cluster ID: 81d3af9e381d46f9b71903b3a5925955
I20260929 15:57:53.782436 8729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:53.815987 8729 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:53.818217 8729 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:53.828194 8729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0be1447bee047408525825c73a67600: Generated new TSK 0
I20260929 15:57:53.829564 8729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:53.896752 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:54.294306 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:54.294632 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:54.294799 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:54.301997 8741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:54.302119 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:54.302302 8741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:54.311949 8741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:54.313426 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:54.314987 8741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:54.329735 8749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.330017 8746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.329969 8747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:54.331651 8741 server_base.cc:1034] running on GCE node
I20260929 15:57:54.332525 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:54.333777 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:54.335028 8741 hybrid_clock.cc:648] HybridClock initialized: now 1790697474334976 us; error 48 us; skew 500 ppm
I20260929 15:57:54.335414 8741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:54 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697474, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:54.339519 8741 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:54.341279 8741 webserver.cc:460] Webserver started at http://127.5.117.1:40399/ using document root <none> and password file <none>
I20260929 15:57:54.342077 8741 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:54.342211 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:54.342623 8741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:54.345927 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "fd793f157ea94696b2e29b9e1e58811e"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "b23500299bd25f99bd2a60717d7b73d5"
server_key_iv: "e5d0a89231ed93cc7b819d7d45c6deb4"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.346784 8741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "fd793f157ea94696b2e29b9e1e58811e"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "b23500299bd25f99bd2a60717d7b73d5"
server_key_iv: "e5d0a89231ed93cc7b819d7d45c6deb4"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.352630 8741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 15:57:54.356312 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.357678 8741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:57:54.357851 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "fd793f157ea94696b2e29b9e1e58811e"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "b23500299bd25f99bd2a60717d7b73d5"
server_key_iv: "e5d0a89231ed93cc7b819d7d45c6deb4"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.358052 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:54.379346 8741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:54.384275 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:54.384634 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:54.385782 8741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:54.387411 8741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:54.387493 8741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.387641 8741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:54.387712 8741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.426211 8741 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:42757
I20260929 15:57:54.426259 8868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:42757 every 8 connection(s)
I20260929 15:57:54.427762 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:54.433146 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8741
I20260929 15:57:54.433394 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:54.434547 5588 external_mini_cluster.cc:1428] Setting key 981f2a03b1f875b397004a5b575159ff
Sep 29 15:57:54 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697474, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:54.455744 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.430878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:35961 (local address 127.5.117.62:41845)
0929 15:57:54.431407 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.431414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.433114 (+ 1700us) server_negotiation.cc:407] Connection header received
0929 15:57:54.436000 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.436005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.436118 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.436245 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.438379 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:54.439202 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:54.442031 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:54.442464 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:54.442588 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:54.447698 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.447707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.447715 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:54.447769 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.451968 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:54.452924 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:54.452942 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:54.452949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:54.453117 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:54.453547 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:54.453555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:54.453561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:54.453988 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.454164 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.454972 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.455243 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":246,"threads_started":1}
I20260929 15:57:54.456775 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.431547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41845 (local address 127.5.117.1:35961)
0929 15:57:54.432837 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.432922 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.435702 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.436503 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.436512 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.437143 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.438088 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:54.438117 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:54.439400 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:54.439408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:54.441822 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:54.441842 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:54.442739 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:54.442748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:54.443136 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:54.444024 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:54.444107 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.447433 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.452214 (+ 4781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:54.452222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:54.452240 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:54.452761 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:54.453232 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:54.453235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:54.453237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:54.453421 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:54.454122 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.454128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.454533 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.454803 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.455296 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":479,"threads_started":1}
I20260929 15:57:54.458729 8869 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41845
I20260929 15:57:54.459082 8869 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 15:57:54.459937 8869 heartbeater.cc:511] Master 127.5.117.62:41845 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
I20260929 15:57:54.462889 8680 ts_manager.cc:194] Registered new tserver with Master: fd793f157ea94696b2e29b9e1e58811e (127.5.117.1:42757)
I20260929 15:57:54.464679 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:35961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:54.489820 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:54.863427 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:54.863744 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:54.863847 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:54.871258 8875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:54.871368 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:54.871492 8875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:54.882045 8875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:54.883726 8875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:54.885239 8875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:54.899349 8880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.899433 8881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.899554 8883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:54.900395 8875 server_base.cc:1034] running on GCE node
I20260929 15:57:54.901006 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:54.901981 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:54.903208 8875 hybrid_clock.cc:648] HybridClock initialized: now 1790697474903174 us; error 45 us; skew 500 ppm
I20260929 15:57:54.903523 8875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:54 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697474, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:54.911206 8875 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:54.912958 8875 webserver.cc:460] Webserver started at http://127.5.117.2:38721/ using document root <none> and password file <none>
I20260929 15:57:54.913677 8875 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:54.913782 8875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:54.914059 8875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:54.917207 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "a6dccf4b51114a5fac5e79336e2c3ece"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "2230aa5d93be17c17236d0541d9d2a8e"
server_key_iv: "62e37c607670fbd8da288921bc2afaa0"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.917979 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "a6dccf4b51114a5fac5e79336e2c3ece"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "2230aa5d93be17c17236d0541d9d2a8e"
server_key_iv: "62e37c607670fbd8da288921bc2afaa0"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.923321 8875 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:54.927220 8890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.928756 8875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:54.928941 8875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "a6dccf4b51114a5fac5e79336e2c3ece"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-b69v"
server_key: "2230aa5d93be17c17236d0541d9d2a8e"
server_key_iv: "62e37c607670fbd8da288921bc2afaa0"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.929147 8875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:54.955562 8875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:54.960511 8875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:54.960901 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:54.962083 8875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:54.963663 8875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:54.963749 8875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.963896 8875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:54.963948 8875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:55.001720 8875 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:34303
I20260929 15:57:55.001828 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:34303 every 8 connection(s)
I20260929 15:57:55.003247 8875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:55.004511 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 8875
I20260929 15:57:55.004726 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:55.005234 5588 external_mini_cluster.cc:1428] Setting key 081a8077b9943deb581cfa7e37b700a4
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697474, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:55.025075 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.006362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:45487 (local address 127.5.117.62:41845)
0929 15:57:55.006901 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.006908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.008163 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 15:57:55.010135 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.010140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.010249 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.010366 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.011913 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.012696 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.014331 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.014743 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.014870 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.018789 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.018794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.018797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.018831 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.021458 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.022242 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.022250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.022255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.022435 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.022922 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.022931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.022937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.023363 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.023546 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.024281 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.024567 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":270,"threads_started":1}
I20260929 15:57:55.025888 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.007054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41845 (local address 127.5.117.2:45487)
0929 15:57:55.007966 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:55.008011 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:55.009863 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:55.010609 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:55.010617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:55.011139 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:55.011751 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.011768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.012850 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.012853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.014153 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.014177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.014982 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.014992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.015244 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.016013 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:55.016038 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:55.018548 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:55.021595 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:55.021604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:55.021621 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:55.022081 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:55.022547 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:55.022550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:55.022552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:55.022788 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:55.023498 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:55.023505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:55.023851 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 15:57:55.024172 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 15:57:55.024604 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":393,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:57:55.027575 9003 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41845
I20260929 15:57:55.027921 9003 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:55.028735 9003 heartbeater.cc:511] Master 127.5.117.62:41845 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
I20260929 15:57:55.030431 8680 ts_manager.cc:194] Registered new tserver with Master: a6dccf4b51114a5fac5e79336e2c3ece (127.5.117.2:34303)
I20260929 15:57:55.031841 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:45487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:55.055562 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:55.425168 9009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:55.425506 9009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:55.425683 9009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:55.433631 9009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:55.433784 9009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:55.433996 9009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:55.444274 9009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:55.445751 9009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:55.447232 9009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:55.463852 9015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:55.463733 9014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:55.463774 9017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:55.464895 9009 server_base.cc:1034] running on GCE node
I20260929 15:57:55.465767 9009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:55.466872 9009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:55.468160 9009 hybrid_clock.cc:648] HybridClock initialized: now 1790697475468094 us; error 59 us; skew 500 ppm
I20260929 15:57:55.468406 8869 heartbeater.cc:503] Master 127.5.117.62:41845 was elected leader, sending a full tablet report...
I20260929 15:57:55.468547 9009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:55.472461 9009 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:55.474116 9009 webserver.cc:460] Webserver started at http://127.5.117.3:41595/ using document root <none> and password file <none>
I20260929 15:57:55.474944 9009 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:55.475098 9009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:55.475435 9009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:55.479080 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "ffd2eb8e8456465c80810ad21211d562"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-b69v"
server_key: "63b467a2cc7032e1fcfc7714f5fc0b6b"
server_key_iv: "a9330c7f5ad71c6f0cc788d05d1dd7f6"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.479985 9009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "ffd2eb8e8456465c80810ad21211d562"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-b69v"
server_key: "63b467a2cc7032e1fcfc7714f5fc0b6b"
server_key_iv: "a9330c7f5ad71c6f0cc788d05d1dd7f6"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.485807 9009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:55.489871 9024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:55.491675 9009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 15:57:55.491878 9009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "ffd2eb8e8456465c80810ad21211d562"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-b69v"
server_key: "63b467a2cc7032e1fcfc7714f5fc0b6b"
server_key_iv: "a9330c7f5ad71c6f0cc788d05d1dd7f6"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.492161 9009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:55.512324 9009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:55.516752 9009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:55.517059 9009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:55.518123 9009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:55.519541 9009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:55.519644 9009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:55.519781 9009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:55.519850 9009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:55.553408 9009 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:46619
I20260929 15:57:55.553455 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:46619 every 8 connection(s)
I20260929 15:57:55.554953 9009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:55.562882 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9009
I20260929 15:57:55.563082 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:55.563714 5588 external_mini_cluster.cc:1428] Setting key 499e4d88e65a18cbd6d65d3edfd62141
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:55.576149 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.557861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:57713 (local address 127.5.117.62:41845)
0929 15:57:55.558318 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.558323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.559340 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 15:57:55.561447 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.561453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.561593 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.561717 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.563654 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.564604 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.566603 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.567144 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.567227 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.570583 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.570588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.570593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.570629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.572957 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.573711 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.573718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.573722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.573856 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.574310 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.574316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.574319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.574599 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.574757 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.575435 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.575681 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":206,"threads_started":1}
I20260929 15:57:55.577185 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.558433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41845 (local address 127.5.117.3:57713)
0929 15:57:55.559191 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:55.559229 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:55.561106 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:55.561976 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:55.561984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:55.562680 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:55.563494 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.563519 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.564752 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.564757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.566416 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.566434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.567334 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.567339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.567511 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.568286 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:55.568313 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:55.570407 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:55.573111 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:55.573120 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:55.573136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:55.573587 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:55.573983 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:55.573989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:55.573996 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:55.574197 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:55.574718 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:55.574727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:55.575083 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 15:57:55.575359 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 15:57:55.575919 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":297,"threads_started":1}
I20260929 15:57:55.578584 9137 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41845
I20260929 15:57:55.578962 9137 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:55.579783 9137 heartbeater.cc:511] Master 127.5.117.62:41845 requested a full tablet report, sending...
I20260929 15:57:55.581225 8680 ts_manager.cc:194] Registered new tserver with Master: ffd2eb8e8456465c80810ad21211d562 (127.5.117.3:46619)
I20260929 15:57:55.582067 8680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:57713
I20260929 15:57:55.590801 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:55.645877 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.629536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.5.117.62:41845)
0929 15:57:55.629822 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.629830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.630064 (+ 234us) server_negotiation.cc:407] Connection header received
0929 15:57:55.630330 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.630334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.630443 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.630557 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.631950 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.632869 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.634221 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.634691 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.634840 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.639632 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.639655 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.639662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.639723 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.642692 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.643516 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.643523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.643527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.643660 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.644042 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.644073 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.644076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.644568 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.644896 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.645232 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.645512 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 15:57:55.646549 8680 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46518
I20260929 15:57:55.648268 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8741
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:55.665865 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.649438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.5.117.1:42757)
0929 15:57:55.650026 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.650035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.650061 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:57:55.650137 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.650145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.650389 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.650531 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.651739 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.652882 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.654113 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.654741 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.654853 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.659099 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.659113 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.659128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.659162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.662529 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.663684 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.663693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.663695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.663846 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.664360 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.664364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.664366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.664793 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.665057 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.665283 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.665515 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":271,"threads_started":1}
W20260929 15:57:55.667497 8864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37038
W20260929 15:57:55.668840 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8741: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:55.669049 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8741
I20260929 15:57:55.688009 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8875
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:55.707288 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.689127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.5.117.2:34303)
0929 15:57:55.689785 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.689794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.689824 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:57:55.689900 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.689908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.690180 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.690334 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.691630 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.692802 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.694095 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.694699 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.694800 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.700472 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.700506 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.700526 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.700596 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.704317 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.705079 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.705096 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.705099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.705270 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.705632 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.705635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.705637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.706031 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.706294 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.706691 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.706943 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":335,"threads_started":1}
W20260929 15:57:55.708832 8998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52714
W20260929 15:57:55.710099 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:55.710322 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8875
I20260929 15:57:55.733789 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9009
Sep 29 15:57:55 dist-test-slave-b69v krb5kdc[8635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697475, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:55.753533 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.735089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.5.117.3:46619)
0929 15:57:55.735956 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.735964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.736007 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:57:55.736183 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.736190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.736449 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.736673 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.738208 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.739298 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.740665 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.741240 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.741382 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.746515 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.746526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.746542 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:55.746582 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.750082 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.751005 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.751014 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.751016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.751174 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:55.751551 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:55.751557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:55.751560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:55.751989 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.752426 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.752757 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.753117 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":417,"threads_started":1}
W20260929 15:57:55.755198 9132 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50508
W20260929 15:57:55.756546 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:55.756762 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9009
I20260929 15:57:55.774799 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 8648
I20260929 15:57:55.776189 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:55.894555 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 8648
2026-09-29T15:57:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4038 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:56 dist-test-slave-b69v krb5kdc[9155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:56 dist-test-slave-b69v krb5kdc[9155](info): set up 2 sockets
Sep 29 15:57:56 dist-test-slave-b69v krb5kdc[9155](info): commencing operation
krb5kdc: starting...
W20260929 15:57:57.021067 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:57:57 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:57:57.259842 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39013
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:57.625016 9168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:57.625355 9168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:57.625520 9168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:57.633036 9168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:57.633154 9168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:57.633288 9168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:57.633420 9168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:57.633504 9168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:57.644017 9168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:57.645455 9168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:57.646924 9168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:57.659783 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:57.659780 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:57.659823 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:57.660607 9168 server_base.cc:1034] running on GCE node
I20260929 15:57:57.661520 9168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:57.663090 9168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:57.664331 9168 hybrid_clock.cc:648] HybridClock initialized: now 1790697477664284 us; error 55 us; skew 500 ppm
I20260929 15:57:57.664724 9168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:57 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697477, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:57.668483 9168 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:57:57.670163 9168 webserver.cc:460] Webserver started at http://127.5.117.62:33065/ using document root <none> and password file <none>
I20260929 15:57:57.670969 9168 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:57.671125 9168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:57.671468 9168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:57.674921 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "8f985ffe2691465b8a95fb1b9d238885"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-b69v"
server_key: "d708789a674695f9f21a34c3738eb770"
server_key_iv: "7538645b846cb9e578e2cc498105c2a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.675774 9168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "8f985ffe2691465b8a95fb1b9d238885"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-b69v"
server_key: "d708789a674695f9f21a34c3738eb770"
server_key_iv: "7538645b846cb9e578e2cc498105c2a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.681324 9168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 15:57:57.685317 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:57.686761 9168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:57:57.686956 9168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "8f985ffe2691465b8a95fb1b9d238885"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-b69v"
server_key: "d708789a674695f9f21a34c3738eb770"
server_key_iv: "7538645b846cb9e578e2cc498105c2a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.687167 9168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:57.704610 9168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:57.709586 9168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:57.710035 9168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:57.731868 9168 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39013
I20260929 15:57:57.731894 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39013 every 8 connection(s)
I20260929 15:57:57.733566 9168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:57.738289 9235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:57.742879 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9168
I20260929 15:57:57.743137 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:57:57.743892 5588 external_mini_cluster.cc:1428] Setting key fd2252b04d6cbfd3d8301ee959a49d5a
I20260929 15:57:57.750146 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885: Bootstrap starting.
I20260929 15:57:57.754138 9235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:57.755398 9235 log.cc:824] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:57.758239 9235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885: No bootstrap required, opened a new log
Sep 29 15:57:57 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:57.768064 9235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } }
I20260929 15:57:57.768558 9235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:57.768697 9235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f985ffe2691465b8a95fb1b9d238885, State: Initialized, Role: FOLLOWER
I20260929 15:57:57.769330 9235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } }
I20260929 15:57:57.769605 9235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:57.769757 9235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:57.769848 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:57.746065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.5.117.62:39013)
0929 15:57:57.746960 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:57.746972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:57.747018 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:57:57.748954 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:57.748977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:57.749351 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:57.749690 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:57.751586 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.753188 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.754940 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.755700 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.755863 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.760872 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:57.760892 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:57.760919 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:57.761001 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:57.765372 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:57.766242 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:57.766249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:57.766256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:57.766442 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:57.766837 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:57.766840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:57.766842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:57.767395 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:57.767623 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:57.768140 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 15:57:57.768646 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":396,"threads_started":1}
I20260929 15:57:57.769992 9235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:57.772868 9235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } }
I20260929 15:57:57.773407 9235 leader_election.cc:302] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f985ffe2691465b8a95fb1b9d238885; no voters:
I20260929 15:57:57.773998 9235 leader_election.cc:288] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:57.774154 9240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:57.774871 9240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [term 1 LEADER]: Becoming Leader. State: Replica: 8f985ffe2691465b8a95fb1b9d238885, State: Running, Role: LEADER
I20260929 15:57:57.775444 9240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } }
I20260929 15:57:57.776111 9235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:57.778748 9242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f985ffe2691465b8a95fb1b9d238885. Latest consensus state: current_term: 1 leader_uuid: "8f985ffe2691465b8a95fb1b9d238885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } } }
I20260929 15:57:57.778788 9241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f985ffe2691465b8a95fb1b9d238885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f985ffe2691465b8a95fb1b9d238885" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39013 } } }
I20260929 15:57:57.779299 9242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:57.779347 9241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:57.780378 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:57.789945 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:57.803716 9249 catalog_manager.cc:1347] Generated new cluster ID: affafa1702834628bd34704ca21bf339
I20260929 15:57:57.803862 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:57.820921 9249 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:57.822616 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:57.832330 9249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f985ffe2691465b8a95fb1b9d238885: Generated new TSK 0
I20260929 15:57:57.833305 9249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:57:57.893211 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:58.314862 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:58.315198 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:58.315325 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:58.324234 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:58.324407 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:58.324550 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:57:58.337669 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:58.339416 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:58.341347 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:58.357982 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:58.358068 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:58.358058 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:58.359287 9261 server_base.cc:1034] running on GCE node
I20260929 15:57:58.360338 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:58.361546 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:58.362807 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790697478362733 us; error 62 us; skew 500 ppm
I20260929 15:57:58.363185 9261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:58 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:58.366946 9261 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:57:58.368857 9261 webserver.cc:460] Webserver started at http://127.5.117.1:39685/ using document root <none> and password file <none>
I20260929 15:57:58.369679 9261 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:58.369831 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:58.370193 9261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:58.373651 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "5bf5a4e0c5dc4fa7aefed2086af1c70b"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "cda11e3e62007aa565293438a5479a5b"
server_key_iv: "8eb16923d0d8313b6395081f76549a17"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.374543 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf5a4e0c5dc4fa7aefed2086af1c70b"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "cda11e3e62007aa565293438a5479a5b"
server_key_iv: "8eb16923d0d8313b6395081f76549a17"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.382259 9261 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:57:58.386425 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.388490 9261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:57:58.388675 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "5bf5a4e0c5dc4fa7aefed2086af1c70b"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "cda11e3e62007aa565293438a5479a5b"
server_key_iv: "8eb16923d0d8313b6395081f76549a17"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.388909 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:58.403963 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:58.409014 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:58.409379 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:58.410496 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:58.412242 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:58.412352 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.412508 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:58.412580 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.451427 9261 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36895
I20260929 15:57:58.451469 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36895 every 8 connection(s)
I20260929 15:57:58.452904 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:58.459380 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9261
I20260929 15:57:58.459905 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:58.460917 5588 external_mini_cluster.cc:1428] Setting key e78b3414482a508f4f031e128f6db071
Sep 29 15:57:58 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:58.476840 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:58.455764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:52809 (local address 127.5.117.62:39013)
0929 15:57:58.456323 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:58.456328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:58.457264 (+ 936us) server_negotiation.cc:407] Connection header received
0929 15:57:58.459447 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:58.459452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:58.459643 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:58.459741 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:58.462606 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.463702 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.465884 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.466332 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.466480 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:58.470532 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:58.470538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:58.470542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:58.470574 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:58.473349 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:58.474326 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:58.474331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:58.474334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:58.474452 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:58.474954 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:58.474958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:58.474960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:58.475233 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:58.475348 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:58.476251 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 15:57:58.476428 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":429,"thread_start_us":302,"threads_started":1}
I20260929 15:57:58.478379 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:58.456340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39013 (local address 127.5.117.1:52809)
0929 15:57:58.457093 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:58.457143 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:58.459095 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:58.460206 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:58.460218 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:58.461080 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:58.462328 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:58.462372 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.463984 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.463994 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:58.465702 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:58.465719 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.466577 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.466586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:58.466846 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:58.467625 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:58.467648 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:58.470300 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:58.473495 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:58.473508 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:58.473524 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:58.474185 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:58.474571 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:58.474586 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:58.474593 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:58.474843 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:58.475356 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:58.475366 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:58.475804 (+ 438us) client_negotiation.cc:769] Sending connection context
0929 15:57:58.476141 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 15:57:58.476763 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":294,"threads_started":1}
I20260929 15:57:58.480438 9389 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39013
I20260929 15:57:58.480765 9389 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:58.481638 9389 heartbeater.cc:511] Master 127.5.117.62:39013 requested a full tablet report, sending...
I20260929 15:57:58.483794 9200 ts_manager.cc:194] Registered new tserver with Master: 5bf5a4e0c5dc4fa7aefed2086af1c70b (127.5.117.1:36895)
I20260929 15:57:58.485399 9200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:52809
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:57:58.510723 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:58.838769 9395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:58.839095 9395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:58.839252 9395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:58.845618 9395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:58.845754 9395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:58.845933 9395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:57:58.855178 9395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:58.856684 9395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:58.858107 9395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:58.875799 9400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:58.875833 9403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:58.875860 9401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:58.877018 9395 server_base.cc:1034] running on GCE node
I20260929 15:57:58.877651 9395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:58.878546 9395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:58.879781 9395 hybrid_clock.cc:648] HybridClock initialized: now 1790697478879732 us; error 51 us; skew 500 ppm
I20260929 15:57:58.880191 9395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:58 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:58.883376 9395 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:57:58.884830 9395 webserver.cc:460] Webserver started at http://127.5.117.2:45289/ using document root <none> and password file <none>
I20260929 15:57:58.885542 9395 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:58.885676 9395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:58.886023 9395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:58.888990 9395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "b88d1496dab145429950a95c2d4914d4"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "079b8dc88f6e36a117101766feaa7ec8"
server_key_iv: "526ebc25b71a44e0e11f6fe6beab476b"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.889745 9395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "b88d1496dab145429950a95c2d4914d4"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "079b8dc88f6e36a117101766feaa7ec8"
server_key_iv: "526ebc25b71a44e0e11f6fe6beab476b"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.895219 9395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:58.898859 9410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.900238 9395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:58.900388 9395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "b88d1496dab145429950a95c2d4914d4"
format_stamp: "Formatted at 2026-09-29 15:57:58 on dist-test-slave-b69v"
server_key: "079b8dc88f6e36a117101766feaa7ec8"
server_key_iv: "526ebc25b71a44e0e11f6fe6beab476b"
server_key_version: "encryptionkey@0"
I20260929 15:57:58.900586 9395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:58.923295 9395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:58.927632 9395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:58.927909 9395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:58.928853 9395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:58.930190 9395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:58.930262 9395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.930380 9395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:58.930429 9395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:58.964790 9395 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:37331
I20260929 15:57:58.964881 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:37331 every 8 connection(s)
I20260929 15:57:58.966251 9395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:58.967206 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9395
I20260929 15:57:58.967437 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:58.968006 5588 external_mini_cluster.cc:1428] Setting key 2db1a7e2a5441c8b3d3a3d4cd48054e2
Sep 29 15:57:58 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:58.990087 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:58.969344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:51505 (local address 127.5.117.62:39013)
0929 15:57:58.969585 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:58.969603 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:58.970993 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 15:57:58.974075 (+ 3082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:58.974079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:58.974161 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:58.974257 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:58.976496 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.977298 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.979498 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.980277 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.980423 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:58.984310 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:58.984315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:58.984319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:58.984348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:58.986897 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:58.987701 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:58.987705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:58.987707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:58.987831 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:58.988309 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:58.988314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:58.988317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:58.988592 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:58.988774 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:58.989440 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 15:57:58.989694 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:57:58.991180 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:58.969928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39013 (local address 127.5.117.2:51505)
0929 15:57:58.970796 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:58.970870 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:58.973777 (+ 2907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:58.974524 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:58.974536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:58.975336 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:58.976314 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:58.976347 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.977435 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.977442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:58.979361 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:58.979373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:58.980483 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:58.980487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:58.980730 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:58.981588 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:58.981613 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:58.984111 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:58.987026 (+ 2915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:58.987034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:58.987069 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:58.987577 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:58.987928 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:58.987934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:58.987937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:58.988188 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:58.988725 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:58.988733 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:58.989087 (+ 354us) client_negotiation.cc:769] Sending connection context
0929 15:57:58.989333 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 15:57:58.989912 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:57:58.992616 9523 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39013
I20260929 15:57:58.993021 9523 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
I20260929 15:57:58.994023 9523 heartbeater.cc:511] Master 127.5.117.62:39013 requested a full tablet report, sending...
I20260929 15:57:58.995692 9200 ts_manager.cc:194] Registered new tserver with Master: b88d1496dab145429950a95c2d4914d4 (127.5.117.2:37331)
I20260929 15:57:58.997038 9200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:51505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:57:59.016677 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:59.376770 9528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:59.377121 9528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:59.377245 9528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:59.384469 9528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:59.384577 9528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:59.384691 9528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:57:59.394564 9528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:57:59.395880 9528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:59.397290 9528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:59.411377 9533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:59.411337 9536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:59.411432 9534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:59.412346 9528 server_base.cc:1034] running on GCE node
I20260929 15:57:59.412964 9528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:59.413887 9528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:59.415117 9528 hybrid_clock.cc:648] HybridClock initialized: now 1790697479415063 us; error 65 us; skew 500 ppm
I20260929 15:57:59.415833 9528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:59.419644 9528 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:57:59.421227 9528 webserver.cc:460] Webserver started at http://127.5.117.3:44845/ using document root <none> and password file <none>
I20260929 15:57:59.421944 9528 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:59.422093 9528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:59.422436 9528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:59.425565 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "ca9c3ea99a184913ad0f92fb7b649258"
format_stamp: "Formatted at 2026-09-29 15:57:59 on dist-test-slave-b69v"
server_key: "0521fd52a8119acc9075e640c28c97f7"
server_key_iv: "57262f3ae71f2e41c06c5c22113c1372"
server_key_version: "encryptionkey@0"
I20260929 15:57:59.426391 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "ca9c3ea99a184913ad0f92fb7b649258"
format_stamp: "Formatted at 2026-09-29 15:57:59 on dist-test-slave-b69v"
server_key: "0521fd52a8119acc9075e640c28c97f7"
server_key_iv: "57262f3ae71f2e41c06c5c22113c1372"
server_key_version: "encryptionkey@0"
I20260929 15:57:59.432164 9528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 15:57:59.436147 9543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:59.437929 9528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:59.438249 9528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "ca9c3ea99a184913ad0f92fb7b649258"
format_stamp: "Formatted at 2026-09-29 15:57:59 on dist-test-slave-b69v"
server_key: "0521fd52a8119acc9075e640c28c97f7"
server_key_iv: "57262f3ae71f2e41c06c5c22113c1372"
server_key_version: "encryptionkey@0"
I20260929 15:57:59.438586 9528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:59.468806 9528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:59.474247 9528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:59.474615 9528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:59.475747 9528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:59.477478 9528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:59.477586 9528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:59.477751 9528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:59.477830 9528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:59.488850 9389 heartbeater.cc:503] Master 127.5.117.62:39013 was elected leader, sending a full tablet report...
I20260929 15:57:59.520669 9528 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:46825
I20260929 15:57:59.520720 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:46825 every 8 connection(s)
I20260929 15:57:59.522120 9528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:59.531709 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9528
I20260929 15:57:59.531919 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:59.532405 5588 external_mini_cluster.cc:1428] Setting key 2f0bd778823bb0e6ba5fcc6ae8a6bddd
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:59.543002 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.525318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36833 (local address 127.5.117.62:39013)
0929 15:57:59.526218 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.526223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.526989 (+ 766us) server_negotiation.cc:407] Connection header received
0929 15:57:59.529489 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.529502 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.529595 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.529714 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.531175 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.532015 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.533692 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.534155 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.534291 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.537712 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.537717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.537720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.537748 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.540015 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.540742 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.540748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.540752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.540878 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.541292 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.541299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.541304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.541577 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.541729 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.542350 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.542577 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":652,"threads_started":1}
I20260929 15:57:59.543905 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.526068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39013 (local address 127.5.117.3:36833)
0929 15:57:59.526833 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:59.526871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:59.529236 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:59.529935 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:59.529943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:59.530428 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:59.531018 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:59.531035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.532163 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.532172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:59.533543 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:59.533556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.534371 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.534376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:59.534584 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.535308 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:59.535329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:59.537528 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:59.540181 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:59.540190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:59.540204 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:59.540626 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:59.540983 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:59.540997 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:59.541000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:59.541192 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:59.541695 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:59.541703 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:59.542007 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 15:57:59.542260 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 15:57:59.542736 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":298,"threads_started":1}
I20260929 15:57:59.545048 9656 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39013
I20260929 15:57:59.545337 9656 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:59.546217 9656 heartbeater.cc:511] Master 127.5.117.62:39013 requested a full tablet report, sending...
I20260929 15:57:59.547703 9200 ts_manager.cc:194] Registered new tserver with Master: ca9c3ea99a184913ad0f92fb7b649258 (127.5.117.3:46825)
I20260929 15:57:59.548506 9200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:36833
I20260929 15:57:59.559319 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:57:59.609279 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.594637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.5.117.1:36895)
0929 15:57:59.595390 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.595398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.595428 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:57:59.595506 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.595513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.595783 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.595924 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.597291 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.598267 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.599330 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.599853 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.599941 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.603632 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.603643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.603657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.603688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.606537 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.607123 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.607136 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.607138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.607254 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.607620 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.607625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.607629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.607959 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.608285 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.608609 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.608896 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":430,"threads_started":1}
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:57:59.628475 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.613728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.5.117.62:39013)
0929 15:57:59.613926 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.613931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.614079 (+ 148us) server_negotiation.cc:407] Connection header received
0929 15:57:59.614180 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.614184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.614272 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.614372 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.615605 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.616466 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.617738 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.618222 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.618331 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.622640 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.622645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.622649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.622688 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.625689 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.626342 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.626347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.626349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.626496 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.626890 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.626894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.626897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.627288 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.627490 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.627922 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.628203 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 15:57:59.629246 9200 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47998
I20260929 15:57:59.630570 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9261
I20260929 15:57:59.644654 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.631653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.5.117.1:36895)
0929 15:57:59.631949 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.631955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.632423 (+ 468us) server_negotiation.cc:407] Connection header received
0929 15:57:59.632715 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.632720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.632794 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.632892 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.634326 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.635244 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.636640 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.637092 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.637208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.639313 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.639318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.639322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.639353 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.642206 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.642837 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.642841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.642844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.642986 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.643351 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.643354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.643356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.643620 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.643806 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.644036 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.644290 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 15:57:59.645329 9384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:59.734411 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9261
I20260929 15:57:59.754797 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9395
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:57:59.772414 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.755831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.5.117.2:37331)
0929 15:57:59.756490 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.756499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.756530 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:57:59.756604 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.756611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.756877 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.757028 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.758401 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.759520 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.760891 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.761534 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.761639 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.765905 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.765916 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.765930 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.765984 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.769321 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.770412 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.770420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.770422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.770563 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.770956 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.770959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.770961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.771323 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.771576 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.771812 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.772020 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":364,"threads_started":1}
I20260929 15:57:59.773883 9518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:59.863963 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9395
I20260929 15:57:59.883602 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9528
Sep 29 15:57:59 dist-test-slave-b69v krb5kdc[9155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:57:59.900807 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:59.884872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.5.117.3:46825)
0929 15:57:59.885633 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:59.885641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:59.885683 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:57:59.885759 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:59.885766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:59.886036 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:59.886192 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:59.887570 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.888793 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.889988 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:59.890541 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:59.890652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:59.895054 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:59.895065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:59.895082 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:59.895157 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:59.898363 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.898920 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.898927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.898930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.899038 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:59.899351 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:59.899354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:59.899356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:59.899714 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:59.899960 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:59.900200 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:57:59.900404 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":379,"threads_started":1}
I20260929 15:57:59.902324 9651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:59.989763 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9528
I20260929 15:58:00.010658 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9168
I20260929 15:58:00.012223 9230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:00.134570 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9168
2026-09-29T15:58:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4241 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:00 dist-test-slave-b69v krb5kdc[9676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:00 dist-test-slave-b69v krb5kdc[9676](info): set up 2 sockets
Sep 29 15:58:00 dist-test-slave-b69v krb5kdc[9676](info): commencing operation
krb5kdc: starting...
W20260929 15:58:01.252550 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:01 dist-test-slave-b69v krb5kdc[9676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:58:01.452363 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40667
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:01.818398 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:01.818755 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:01.818933 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:01.826097 9689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:01.826231 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:01.826331 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:01.826433 9689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:01.826522 9689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:01.836061 9689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:01.837400 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:01.838804 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:01.851897 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:01.852018 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:01.852545 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:01.853066 9689 server_base.cc:1034] running on GCE node
I20260929 15:58:01.853884 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:01.855315 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:01.856616 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790697481856504 us; error 84 us; skew 500 ppm
I20260929 15:58:01.857121 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:01 dist-test-slave-b69v krb5kdc[9676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:01.860443 9689 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:58:01.861621 9689 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:01.861757 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:01.862089 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:01.865154 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "64ece5f4832648cca78b5c4db3df9ce8"
format_stamp: "Formatted at 2026-09-29 15:58:01 on dist-test-slave-b69v"
server_key: "2f36cbdd5477a426726193d5fd447a8f"
server_key_iv: "a9aec20d557465e31341ce68273a374b"
server_key_version: "encryptionkey@0"
I20260929 15:58:01.865990 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "64ece5f4832648cca78b5c4db3df9ce8"
format_stamp: "Formatted at 2026-09-29 15:58:01 on dist-test-slave-b69v"
server_key: "2f36cbdd5477a426726193d5fd447a8f"
server_key_iv: "a9aec20d557465e31341ce68273a374b"
server_key_version: "encryptionkey@0"
I20260929 15:58:01.870988 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:01.874439 9703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:01.875643 9689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:58:01.875837 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "64ece5f4832648cca78b5c4db3df9ce8"
format_stamp: "Formatted at 2026-09-29 15:58:01 on dist-test-slave-b69v"
server_key: "2f36cbdd5477a426726193d5fd447a8f"
server_key_iv: "a9aec20d557465e31341ce68273a374b"
server_key_version: "encryptionkey@0"
I20260929 15:58:01.876096 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:01.893342 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:01.898142 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:01.898452 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:01.919533 9689 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:40667
I20260929 15:58:01.919561 9754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:40667 every 8 connection(s)
I20260929 15:58:01.920547 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:01.923537 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9689
I20260929 15:58:01.923808 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:01.924566 5588 external_mini_cluster.cc:1428] Setting key 051ce1f77e5d8e0c584bb9ffd76e50a5
I20260929 15:58:01.924922 9755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:01.939708 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: Bootstrap starting.
Sep 29 15:58:01 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:01.943784 9755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:01.945284 9755 log.cc:824] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:01.948760 9755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: No bootstrap required, opened a new log
I20260929 15:58:01.950349 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:01.926752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.5.117.62:40667)
0929 15:58:01.927679 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:01.927694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:01.927745 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:58:01.930192 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:01.930222 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:01.930741 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:01.931217 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:01.932908 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:01.934908 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:01.936246 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:01.937373 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:01.937582 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:01.941518 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:01.941533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:01.941556 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:01.941613 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:01.945536 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:01.946442 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:01.946448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:01.946453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:01.946715 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:01.947088 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:01.947092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:01.947095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:01.947807 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:01.948145 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:01.948574 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 15:58:01.948980 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":420,"threads_started":1}
I20260929 15:58:01.957639 9755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } }
I20260929 15:58:01.957950 9755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:01.958029 9755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ece5f4832648cca78b5c4db3df9ce8, State: Initialized, Role: FOLLOWER
I20260929 15:58:01.958516 9755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } }
I20260929 15:58:01.958741 9755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:01.958863 9755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:01.958992 9755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:01.961627 9755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } }
I20260929 15:58:01.962105 9755 leader_election.cc:302] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64ece5f4832648cca78b5c4db3df9ce8; no voters:
I20260929 15:58:01.962781 9755 leader_election.cc:288] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:01.963208 9760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:01.963932 9760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 64ece5f4832648cca78b5c4db3df9ce8, State: Running, Role: LEADER
I20260929 15:58:01.964613 9760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } }
I20260929 15:58:01.965216 9755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:01.967638 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64ece5f4832648cca78b5c4db3df9ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } } }
I20260929 15:58:01.967969 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64ece5f4832648cca78b5c4db3df9ce8. Latest consensus state: current_term: 1 leader_uuid: "64ece5f4832648cca78b5c4db3df9ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ece5f4832648cca78b5c4db3df9ce8" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40667 } } }
I20260929 15:58:01.968215 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:01.968326 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:01.970378 9769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:01.975087 9769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:01.983309 9769 catalog_manager.cc:1347] Generated new cluster ID: 2a611a4e6dea4203a264625086ac8156
I20260929 15:58:01.983409 9769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:01.998446 9769 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:02.000000 9769 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:02.006845 9769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: Generated new TSK 0
I20260929 15:58:02.007941 9769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:58:02.081808 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:02.462095 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:02.462486 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:02.462673 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:02.470837 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:02.471004 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:02.471186 9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:02.482177 9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:02.483855 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:02.485489 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:02.499441 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.499356 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.499406 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:02.500284 9781 server_base.cc:1034] running on GCE node
I20260929 15:58:02.500852 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:02.501863 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:02.503067 9781 hybrid_clock.cc:648] HybridClock initialized: now 1790697482503019 us; error 45 us; skew 500 ppm
I20260929 15:58:02.503412 9781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:02 dist-test-slave-b69v krb5kdc[9676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697482, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:02.506994 9781 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:58:02.508260 9781 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:02.508374 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:02.508715 9781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:02.511709 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1143c54030404215aa2e7078529a5b2e"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-b69v"
server_key: "1abae95082ee7b3262983ee2f1af8660"
server_key_iv: "67eb9a04910765ef3f4ced3c78df1bdb"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.512594 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1143c54030404215aa2e7078529a5b2e"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-b69v"
server_key: "1abae95082ee7b3262983ee2f1af8660"
server_key_iv: "67eb9a04910765ef3f4ced3c78df1bdb"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.517750 9781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:02.521278 9795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.523260 9781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:02.523463 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1143c54030404215aa2e7078529a5b2e"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-b69v"
server_key: "1abae95082ee7b3262983ee2f1af8660"
server_key_iv: "67eb9a04910765ef3f4ced3c78df1bdb"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.523710 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:02.569726 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:02.574295 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:02.574607 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:02.575628 9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:02.577209 9781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:02.577332 9781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.577471 9781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:02.577570 9781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.618160 9781 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:42405
I20260929 15:58:02.618180 9907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:42405 every 8 connection(s)
I20260929 15:58:02.619311 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:02.628511 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9781
I20260929 15:58:02.628796 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:02.629717 5588 external_mini_cluster.cc:1428] Setting key 3090c37aa8c4511848b214c8db85ac4a
Sep 29 15:58:02 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697482, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:02.644567 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:02.622687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:39867 (local address 127.5.117.62:40667)
0929 15:58:02.623356 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:02.623364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:02.624943 (+ 1579us) server_negotiation.cc:407] Connection header received
0929 15:58:02.626906 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:02.626912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:02.626992 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:02.627121 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:02.629104 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.630247 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.632520 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.633218 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.633369 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:02.637308 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:02.637317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:02.637325 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:02.637372 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:02.640946 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.641938 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.641943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.641945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.642076 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.642681 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.642685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.642687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.642968 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:02.643095 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:02.644023 (+ 928us) server_negotiation.cc:299] Negotiation successful
0929 15:58:02.644269 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":311,"threads_started":1}
I20260929 15:58:02.646112 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:02.623822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40667 (local address 127.5.117.1:39867)
0929 15:58:02.624759 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:02.624803 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:02.626599 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:02.627342 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:02.627350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:02.628040 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:02.628925 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:02.628945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.630434 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.630439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:02.632342 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:02.632355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.633447 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.633452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:02.633657 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:02.634451 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:02.634479 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:02.637053 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:02.641095 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:02.641106 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:02.641137 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:02.641785 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:02.642190 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:02.642197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:02.642204 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:02.642440 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:02.643113 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:02.643125 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:02.643587 (+ 462us) client_negotiation.cc:769] Sending connection context
0929 15:58:02.643908 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 15:58:02.644535 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":425,"threads_started":1}
I20260929 15:58:02.647792 9908 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40667
I20260929 15:58:02.648173 9908 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:02.648895 9908 heartbeater.cc:511] Master 127.5.117.62:40667 requested a full tablet report, sending...
I20260929 15:58:02.651139 9720 ts_manager.cc:194] Registered new tserver with Master: 1143c54030404215aa2e7078529a5b2e (127.5.117.1:42405)
I20260929 15:58:02.652765 9720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:39867
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:58:02.683476 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:03.071520 9914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:03.071826 9914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:03.072017 9914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:03.079556 9914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:03.079674 9914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:03.079932 9914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:03.091141 9914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:03.092824 9914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:03.094491 9914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:03.109597 9919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.109709 9922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.109683 9920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:03.110587 9914 server_base.cc:1034] running on GCE node
I20260929 15:58:03.111225 9914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:03.112300 9914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:03.113511 9914 hybrid_clock.cc:648] HybridClock initialized: now 1790697483113468 us; error 28 us; skew 500 ppm
I20260929 15:58:03.113837 9914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:03 dist-test-slave-b69v krb5kdc[9676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.117394 9914 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:58:03.118671 9914 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:03.118767 9914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:03.119078 9914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:03.122424 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "14b5996e68754777a64a68af0345b249"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "2cf9f9371798f234090e421a01b12669"
server_key_iv: "b94ada7352fe6b16a275788c4cd97e7f"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.123291 9914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "14b5996e68754777a64a68af0345b249"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "2cf9f9371798f234090e421a01b12669"
server_key_iv: "b94ada7352fe6b16a275788c4cd97e7f"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.129217 9914 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:58:03.132943 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.134451 9914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:03.134687 9914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "14b5996e68754777a64a68af0345b249"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "2cf9f9371798f234090e421a01b12669"
server_key_iv: "b94ada7352fe6b16a275788c4cd97e7f"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.134941 9914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:03.149261 9914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:03.153779 9914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:03.154036 9914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:03.155134 9914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:03.156759 9914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:03.156853 9914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.156945 9914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:03.157004 9914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.192807 9914 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:43493
I20260929 15:58:03.192850 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:43493 every 8 connection(s)
I20260929 15:58:03.193926 9914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:03.196228 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 9914
I20260929 15:58:03.196565 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:03.197199 5588 external_mini_cluster.cc:1428] Setting key 06d3d31d3db2d81e232468302b9b0c43
Sep 29 15:58:03 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:58:03.225167 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.197111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:33715 (local address 127.5.117.62:40667)
0929 15:58:03.208573 (+ 11462us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.208582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.208615 (+ 33us) server_negotiation.cc:407] Connection header received
0929 15:58:03.208742 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.208750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.208862 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.209014 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.210874 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.211952 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.213790 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.214311 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.214458 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.218648 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.218653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.218657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.218694 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.221726 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.222488 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.222495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.222499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.222676 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.223146 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.223152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.223174 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.223487 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.223660 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.224494 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.224745 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11266,"thread_start_us":11095,"threads_started":1}
I20260929 15:58:03.226399 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.197839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40667 (local address 127.5.117.2:33715)
0929 15:58:03.204007 (+ 6168us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.204161 (+ 154us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.207518 (+ 3357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.209259 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.209268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.209926 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:03.210620 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.210639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.212194 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.212198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.213624 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.213637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.214569 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.214576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.214829 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.215657 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:03.215682 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:03.218393 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:03.221878 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.221887 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.221906 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.222366 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.222814 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.222819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.222822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.223023 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.223627 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:03.223635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:03.224025 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.224394 (+ 369us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.224994 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5660,"thread_start_us":5363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
I20260929 15:58:03.228214 10041 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40667
I20260929 15:58:03.228564 10041 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:03.229377 10041 heartbeater.cc:511] Master 127.5.117.62:40667 requested a full tablet report, sending...
I20260929 15:58:03.231300 9720 ts_manager.cc:194] Registered new tserver with Master: 14b5996e68754777a64a68af0345b249 (127.5.117.2:43493)
I20260929 15:58:03.232703 9720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:33715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:58:03.253535 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:03.609215 10047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:03.609639 10047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:03.609767 10047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:03.618567 10047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:03.618835 10047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:03.619107 10047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:03.633421 10047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:03.634927 10047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:03.636480 10047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:03.650831 10052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.650977 10053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.651024 10055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:03.652504 10047 server_base.cc:1034] running on GCE node
I20260929 15:58:03.653270 10047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:03.654407 10047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:03.655692 10047 hybrid_clock.cc:648] HybridClock initialized: now 1790697483655605 us; error 68 us; skew 500 ppm
I20260929 15:58:03.655828 9908 heartbeater.cc:503] Master 127.5.117.62:40667 was elected leader, sending a full tablet report...
I20260929 15:58:03.656149 10047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:03 dist-test-slave-b69v krb5kdc[9676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.660916 10047 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:58:03.662278 10047 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:03.662379 10047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:03.662739 10047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:03.666293 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "4f056fabbe1c4eaeb05f13946d093404"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "65b99a108a161d99376f555cb3f607bb"
server_key_iv: "d6b615a7e13d183a5ec2f827565115ae"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.667232 10047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "4f056fabbe1c4eaeb05f13946d093404"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "65b99a108a161d99376f555cb3f607bb"
server_key_iv: "d6b615a7e13d183a5ec2f827565115ae"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.673600 10047 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:58:03.677503 10061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.678869 10047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:03.679070 10047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "4f056fabbe1c4eaeb05f13946d093404"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-b69v"
server_key: "65b99a108a161d99376f555cb3f607bb"
server_key_iv: "d6b615a7e13d183a5ec2f827565115ae"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.679334 10047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:03.704116 10047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:03.710045 10047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:03.710352 10047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:03.711548 10047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:03.713266 10047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:03.713361 10047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.713440 10047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:03.713508 10047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.754284 10047 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:35835
I20260929 15:58:03.754384 10173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:35835 every 8 connection(s)
I20260929 15:58:03.755389 10047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:03.756026 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10047
I20260929 15:58:03.756314 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:03.756973 5588 external_mini_cluster.cc:1428] Setting key 4f93b03aa03c37b31d457f7699dc2d91
Sep 29 15:58:03 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:03.780191 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.758801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:53475 (local address 127.5.117.62:40667)
0929 15:58:03.759661 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.759668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.761631 (+ 1963us) server_negotiation.cc:407] Connection header received
0929 15:58:03.763685 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.763690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.763779 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.763884 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.765814 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.766758 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.768767 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.769641 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.769762 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.774104 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.774111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.774117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.774164 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.777074 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.777815 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.777819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.777822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.777950 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.778333 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.778336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.778338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.778619 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.778748 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.779582 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.779776 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":329,"threads_started":1}
I20260929 15:58:03.781368 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.759314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40667 (local address 127.5.117.3:53475)
0929 15:58:03.761444 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.761506 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.763418 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.764161 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.764169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.764805 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:03.765605 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.765625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.766966 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.766971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.768479 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.768493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.769881 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.769886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.770140 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.770979 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:03.771005 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:03.773836 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:03.777216 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.777223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.777239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.777700 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.778043 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.778046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.778050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.778229 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.778731 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:03.778738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:03.779188 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.779448 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.779965 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":1559,"threads_started":1}
I20260929 15:58:03.783006 10174 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40667
I20260929 15:58:03.783396 10174 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:03.784400 10174 heartbeater.cc:511] Master 127.5.117.62:40667 requested a full tablet report, sending...
I20260929 15:58:03.786458 9719 ts_manager.cc:194] Registered new tserver with Master: 4f056fabbe1c4eaeb05f13946d093404 (127.5.117.3:35835)
I20260929 15:58:03.787533 9719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:53475
I20260929 15:58:03.799503 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:58:03.819609 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.804762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.5.117.62:40667)
0929 15:58:03.804964 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.804968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.805629 (+ 661us) server_negotiation.cc:407] Connection header received
0929 15:58:03.805865 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.805871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.805971 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.806087 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.807626 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.808556 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.809803 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.810240 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.810373 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.812849 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.812858 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.812866 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.812902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.816122 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.816987 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.816993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.816997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.817155 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.817578 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.817583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.817587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.818167 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.818564 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.818923 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.819244 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:58:03.825035 9719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:03.832918 9719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:03.859350 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.848651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:43493 (local address 127.0.0.1:45044)
0929 15:58:03.850084 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.850102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.850469 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.851550 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.851555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.851579 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:03.851887 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.851895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.854553 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.854560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.856115 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.856130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.858324 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.858331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.858557 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.858623 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.858758 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.858947 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":587,"threads_started":1}
I20260929 15:58:03.859699 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.848551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.5.117.2:43493)
0929 15:58:03.849211 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.849225 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.850208 (+ 983us) server_negotiation.cc:407] Connection header received
0929 15:58:03.850772 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.850781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.851070 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.851259 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:03.852093 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.854391 (+ 2298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.856327 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.858073 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.858888 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.859105 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.859192 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.859379 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":328,"threads_started":1}
I20260929 15:58:03.861037 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.849247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:42405 (local address 127.0.0.1:44684)
0929 15:58:03.849932 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.849982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.851161 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.853243 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.853250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.853287 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:03.853877 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.853900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.856455 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.856460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.858322 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.858337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.860197 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.860202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.860400 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.860442 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.860549 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.860685 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":945,"threads_started":1}
I20260929 15:58:03.861131 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.849939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.5.117.1:42405)
0929 15:58:03.852669 (+ 2730us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.852679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.852708 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:58:03.852807 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.852814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.853027 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.853224 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:03.854131 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.856278 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.858525 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.859971 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.860134 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.860388 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.860602 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.860767 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2522,"thread_start_us":2335,"threads_started":1}
I20260929 15:58:03.864780 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.850211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:35835 (local address 127.0.0.1:45216)
0929 15:58:03.853916 (+ 3705us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.853939 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.854050 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.857358 (+ 3308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.857363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.857388 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:03.857810 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.857820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.859851 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.859856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.862270 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.862282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.863980 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.863986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.864207 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.864238 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.864353 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.864465 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3545,"thread_start_us":3373,"threads_started":1}
I20260929 15:58:03.864981 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.850906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.5.117.3:35835)
0929 15:58:03.856769 (+ 5863us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.856776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.856810 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:58:03.856916 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.856926 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.857125 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.857344 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:03.858040 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.859668 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.862412 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.863778 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.863917 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.864181 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.864391 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.864539 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5644,"thread_start_us":5407,"threads_started":1}
I20260929 15:58:03.864532 9976 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (DEFAULT_TABLE table=test-table [id=98187075a2a24c57ac0c250b99af67b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:03.864886 9843 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (DEFAULT_TABLE table=test-table [id=98187075a2a24c57ac0c250b99af67b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:03.866297 9843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:03.866297 9976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:03.867969 10109 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (DEFAULT_TABLE table=test-table [id=98187075a2a24c57ac0c250b99af67b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:03.869571 10109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:03.883468 10194 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Bootstrap starting.
I20260929 15:58:03.883898 10193 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Bootstrap starting.
I20260929 15:58:03.885710 10195 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Bootstrap starting.
I20260929 15:58:03.888701 10195 tablet_bootstrap.cc:654] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:03.888854 10193 tablet_bootstrap.cc:654] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:03.888898 10194 tablet_bootstrap.cc:654] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:03.889848 10195 log.cc:824] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:03.890213 10194 log.cc:824] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:03.890211 10193 log.cc:824] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:03.893172 10195 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: No bootstrap required, opened a new log
I20260929 15:58:03.893374 10193 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: No bootstrap required, opened a new log
I20260929 15:58:03.893421 10194 tablet_bootstrap.cc:492] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: No bootstrap required, opened a new log
I20260929 15:58:03.893496 10195 ts_tablet_manager.cc:1397] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:58:03.893654 10194 ts_tablet_manager.cc:1397] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:58:03.893654 10193 ts_tablet_manager.cc:1397] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260929 15:58:03.901165 10195 raft_consensus.cc:348] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.901551 10195 raft_consensus.cc:374] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:03.901683 10195 raft_consensus.cc:729] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f056fabbe1c4eaeb05f13946d093404, State: Initialized, Role: FOLLOWER
I20260929 15:58:03.902403 10195 consensus_queue.cc:260] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.903576 10174 heartbeater.cc:503] Master 127.5.117.62:40667 was elected leader, sending a full tablet report...
I20260929 15:58:03.904316 10195 ts_tablet_manager.cc:1428] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 15:58:03.904541 10194 raft_consensus.cc:348] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.904960 10194 raft_consensus.cc:374] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:03.905058 10194 raft_consensus.cc:729] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b5996e68754777a64a68af0345b249, State: Initialized, Role: FOLLOWER
I20260929 15:58:03.905668 10194 consensus_queue.cc:260] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.906827 10041 heartbeater.cc:503] Master 127.5.117.62:40667 was elected leader, sending a full tablet report...
I20260929 15:58:03.907033 10193 raft_consensus.cc:348] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.907465 10193 raft_consensus.cc:374] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:03.907482 10194 ts_tablet_manager.cc:1428] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 15:58:03.907573 10193 raft_consensus.cc:729] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1143c54030404215aa2e7078529a5b2e, State: Initialized, Role: FOLLOWER
I20260929 15:58:03.908494 10193 consensus_queue.cc:260] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:03.911144 10193 ts_tablet_manager.cc:1428] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260929 15:58:03.946095 10042 tablet.cc:2369] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:04.007304 10175 tablet.cc:2369] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:04.122740 9909 tablet.cc:2369] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:04.231163 10200 raft_consensus.cc:482] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:04.231801 10200 raft_consensus.cc:504] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:04.234323 10200 leader_election.cc:288] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f056fabbe1c4eaeb05f13946d093404 (127.5.117.3:35835), 1143c54030404215aa2e7078529a5b2e (127.5.117.1:42405)
I20260929 15:58:04.244032 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.235288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:48451 (local address 127.5.117.1:42405)
0929 15:58:04.235578 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.235588 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.236408 (+ 820us) server_negotiation.cc:407] Connection header received
0929 15:58:04.236630 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.236634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.236714 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.236841 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:04.237847 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.238690 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.241783 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.242733 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.242929 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.243084 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.243534 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.243707 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:58:04.244028 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.235005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:38461 (local address 127.5.117.3:35835)
0929 15:58:04.235289 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.235299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.236344 (+ 1045us) server_negotiation.cc:407] Connection header received
0929 15:58:04.236608 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.236612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.236687 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.236824 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:04.237759 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.238643 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.241796 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.242728 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.242933 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.243067 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.243532 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.243706 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 15:58:04.244580 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.234947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:42405 (local address 127.5.117.2:48451)
0929 15:58:04.236283 (+ 1336us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:04.236300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:04.236474 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:04.236884 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:04.236889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:04.236923 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:04.237606 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.237622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.238893 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.238903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.241490 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.241510 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.243107 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.243115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.243337 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.243373 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 15:58:04.243499 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 15:58:04.243746 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":154,"thread_start_us":662,"threads_started":1}
I20260929 15:58:04.244756 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.234758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:35835 (local address 127.5.117.2:38461)
0929 15:58:04.236196 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:04.236221 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:04.236445 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:04.236853 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:04.236858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:04.236906 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:04.237481 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.237503 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.238940 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.238944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.241432 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.241468 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.243062 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.243070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.243363 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.243387 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:58:04.243500 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 15:58:04.243747 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1246,"mutex_wait_us":84,"thread_start_us":1151,"threads_started":1}
I20260929 15:58:04.244856 10199 raft_consensus.cc:482] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:04.245198 10199 raft_consensus.cc:504] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:04.246392 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "14b5996e68754777a64a68af0345b249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f056fabbe1c4eaeb05f13946d093404" is_pre_election: true
I20260929 15:58:04.246909 10129 raft_consensus.cc:2454] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b5996e68754777a64a68af0345b249 in term 0.
I20260929 15:58:04.246771 9863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "14b5996e68754777a64a68af0345b249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1143c54030404215aa2e7078529a5b2e" is_pre_election: true
I20260929 15:58:04.247357 9863 raft_consensus.cc:2454] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b5996e68754777a64a68af0345b249 in term 0.
I20260929 15:58:04.247714 10199 leader_election.cc:288] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b5996e68754777a64a68af0345b249 (127.5.117.2:43493), 1143c54030404215aa2e7078529a5b2e (127.5.117.1:42405)
I20260929 15:58:04.247768 9931 leader_election.cc:302] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b5996e68754777a64a68af0345b249, 4f056fabbe1c4eaeb05f13946d093404; no voters:
I20260929 15:58:04.248333 10200 raft_consensus.cc:2781] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:04.248502 10200 raft_consensus.cc:482] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:04.248634 10200 raft_consensus.cc:3037] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:04.252213 10200 raft_consensus.cc:504] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:04.252933 10200 leader_election.cc:288] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [CANDIDATE]: Term 1 election: Requested vote from peers 4f056fabbe1c4eaeb05f13946d093404 (127.5.117.3:35835), 1143c54030404215aa2e7078529a5b2e (127.5.117.1:42405)
I20260929 15:58:04.254041 9863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "14b5996e68754777a64a68af0345b249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1143c54030404215aa2e7078529a5b2e"
I20260929 15:58:04.254098 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "14b5996e68754777a64a68af0345b249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f056fabbe1c4eaeb05f13946d093404"
I20260929 15:58:04.254323 10129 raft_consensus.cc:3037] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:04.254354 9863 raft_consensus.cc:3037] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:04.256299 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.248191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:42405 (local address 127.5.117.3:38139)
0929 15:58:04.248561 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:04.248586 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:04.248724 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:04.249249 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:04.249257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:04.249312 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:04.249593 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.249609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.251166 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.251174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.253302 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.253316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.255348 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.255354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.255619 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.255670 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:58:04.255748 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 15:58:04.255906 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260929 15:58:04.258033 10129 raft_consensus.cc:2454] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b5996e68754777a64a68af0345b249 in term 1.
I20260929 15:58:04.258927 9863 raft_consensus.cc:2454] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b5996e68754777a64a68af0345b249 in term 1.
I20260929 15:58:04.259093 9931 leader_election.cc:302] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b5996e68754777a64a68af0345b249, 4f056fabbe1c4eaeb05f13946d093404; no voters:
I20260929 15:58:04.259413 10200 raft_consensus.cc:2781] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:04.259976 10200 raft_consensus.cc:686] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 1 LEADER]: Becoming Leader. State: Replica: 14b5996e68754777a64a68af0345b249, State: Running, Role: LEADER
I20260929 15:58:04.261034 10200 consensus_queue.cc:237] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } }
I20260929 15:58:04.262223 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.248673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:38139 (local address 127.5.117.1:42405)
0929 15:58:04.248882 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.248888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.248907 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:58:04.248982 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.248989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.249091 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.249204 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:04.249760 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.250964 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.253690 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.255156 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.261593 (+ 6437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.261711 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.261766 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.261907 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":70}
I20260929 15:58:04.262737 9863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "4f056fabbe1c4eaeb05f13946d093404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1143c54030404215aa2e7078529a5b2e" is_pre_election: true
I20260929 15:58:04.263048 9863 raft_consensus.cc:2379] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4f056fabbe1c4eaeb05f13946d093404 in current term 1: Already voted for candidate 14b5996e68754777a64a68af0345b249 in this term.
I20260929 15:58:04.263571 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:43493 (local address 127.5.117.3:43475)
0929 15:58:04.256542 (+ 8329us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:04.256564 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:04.256854 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:04.257377 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:04.257382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:04.257407 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:04.257589 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.257598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.259147 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.259152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.261115 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:04.261142 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.262690 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.262697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:04.262910 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.262951 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:58:04.263071 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:58:04.263217 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8218,"mutex_wait_us":113,"thread_start_us":12322,"threads_started":1}
I20260929 15:58:04.263581 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.253529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:43475 (local address 127.5.117.2:43493)
0929 15:58:04.253751 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.253758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.256638 (+ 2880us) server_negotiation.cc:407] Connection header received
0929 15:58:04.257028 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.257035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.257116 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.257189 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:04.257754 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.258816 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.261312 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.262499 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.262649 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.262752 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.263149 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.263315 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:58:04.264786 9996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bfd7bc5c0a142fcb14ea7d791adad9f" candidate_uuid: "4f056fabbe1c4eaeb05f13946d093404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b5996e68754777a64a68af0345b249" is_pre_election: true
I20260929 15:58:04.266031 10064 leader_election.cc:302] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f056fabbe1c4eaeb05f13946d093404; no voters: 1143c54030404215aa2e7078529a5b2e, 14b5996e68754777a64a68af0345b249
I20260929 15:58:04.266443 10199 raft_consensus.cc:2726] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 15:58:04.267851 9719 catalog_manager.cc:5495] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b5996e68754777a64a68af0345b249 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "14b5996e68754777a64a68af0345b249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b5996e68754777a64a68af0345b249" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 43493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:04.316408 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.310314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.5.117.2:43493)
0929 15:58:04.310511 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.310517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.310539 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:04.310707 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.310711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.310834 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.310980 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:04.311361 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.312278 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.313706 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.314380 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.314483 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.314953 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:04.315645 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:04.315805 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.315899 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.315989 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:58:04.326987 10129 raft_consensus.cc:1261] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 1 FOLLOWER]: Refusing update from remote peer 14b5996e68754777a64a68af0345b249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:04.326946 9863 raft_consensus.cc:1261] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 1 FOLLOWER]: Refusing update from remote peer 14b5996e68754777a64a68af0345b249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:04.328166 10200 consensus_queue.cc:1035] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f056fabbe1c4eaeb05f13946d093404" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:04.328513 10205 consensus_queue.cc:1035] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1143c54030404215aa2e7078529a5b2e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:04.359350 9720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:04.359876 9720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:04.370860 9720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64ece5f4832648cca78b5c4db3df9ce8: Sending DeleteTablet for 3 replicas of tablet 9bfd7bc5c0a142fcb14ea7d791adad9f
I20260929 15:58:04.372593 10109 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:04 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45216
I20260929 15:58:04.372596 9843 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:04 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:44684
I20260929 15:58:04.374048 9976 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bfd7bc5c0a142fcb14ea7d791adad9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:04 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45044
I20260929 15:58:04.376392 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9781
I20260929 15:58:04.384949 10218 tablet_replica.cc:330] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: stopping tablet replica
I20260929 15:58:04.385645 10218 raft_consensus.cc:2229] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:04.386379 10218 raft_consensus.cc:2258] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:04.389516 10220 tablet_replica.cc:330] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: stopping tablet replica
I20260929 15:58:04.389983 10220 raft_consensus.cc:2229] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:04.390812 10220 raft_consensus.cc:2258] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:04.391283 10218 ts_tablet_manager.cc:1905] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:04.393435 10220 ts_tablet_manager.cc:1905] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:04.394167 10219 tablet_replica.cc:330] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: stopping tablet replica
I20260929 15:58:04.394645 10219 raft_consensus.cc:2229] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:04.395239 10219 raft_consensus.cc:2258] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:04.397117 10219 ts_tablet_manager.cc:1905] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:04.399837 10218 ts_tablet_manager.cc:1918] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:04.400023 10218 log.cc:1196] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/9bfd7bc5c0a142fcb14ea7d791adad9f
I20260929 15:58:04.400611 10218 ts_tablet_manager.cc:1939] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 4f056fabbe1c4eaeb05f13946d093404: Deleting consensus metadata
I20260929 15:58:04.402823 9706 catalog_manager.cc:4841] TS 4f056fabbe1c4eaeb05f13946d093404 (127.5.117.3:35835): tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (table test-table [id=98187075a2a24c57ac0c250b99af67b3]) successfully deleted
I20260929 15:58:04.404459 10219 ts_tablet_manager.cc:1918] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:04.404634 10219 log.cc:1196] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/9bfd7bc5c0a142fcb14ea7d791adad9f
I20260929 15:58:04.405218 10219 ts_tablet_manager.cc:1939] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 1143c54030404215aa2e7078529a5b2e: Deleting consensus metadata
I20260929 15:58:04.406960 10220 ts_tablet_manager.cc:1918] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:04.407150 10220 log.cc:1196] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/9bfd7bc5c0a142fcb14ea7d791adad9f
I20260929 15:58:04.407672 9707 catalog_manager.cc:4841] TS 1143c54030404215aa2e7078529a5b2e (127.5.117.1:42405): tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (table test-table [id=98187075a2a24c57ac0c250b99af67b3]) successfully deleted
I20260929 15:58:04.407793 10220 ts_tablet_manager.cc:1939] T 9bfd7bc5c0a142fcb14ea7d791adad9f P 14b5996e68754777a64a68af0345b249: Deleting consensus metadata
I20260929 15:58:04.410167 9706 catalog_manager.cc:4841] TS 14b5996e68754777a64a68af0345b249 (127.5.117.2:43493): tablet 9bfd7bc5c0a142fcb14ea7d791adad9f (table test-table [id=98187075a2a24c57ac0c250b99af67b3]) successfully deleted
Sep 29 15:58:04 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:58:04.420600 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.377746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.5.117.1:42405)
0929 15:58:04.377951 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.377959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.396839 (+ 18880us) server_negotiation.cc:407] Connection header received
0929 15:58:04.397203 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.397210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.397331 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.397420 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:04.403177 (+ 5757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.404443 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.407164 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.407872 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.407976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.413196 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:04.413214 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:04.413237 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:04.413293 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:04.417303 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.418127 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.418138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.418142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.418332 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.418699 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.418704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.418707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.419264 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:04.419604 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.419821 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.420160 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:58:04.421422 9903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.546011 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9781
I20260929 15:58:04.567952 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9914
Sep 29 15:58:04 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:58:04.589522 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.569184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.5.117.2:43493)
0929 15:58:04.569438 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.569447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.569490 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:58:04.569677 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.569684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.569799 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.569929 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:04.571436 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.572553 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.573944 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.574476 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.574641 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.580761 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:04.580787 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:04.580806 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:04.580901 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:04.585981 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.587032 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.587040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.587043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.587233 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.587670 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.587674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.587676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.588237 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:04.588503 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.588935 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.589155 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:04.590456 10036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.721345 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9914
I20260929 15:58:04.742473 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10047
Sep 29 15:58:04 dist-test-slave-b69v krb5kdc[9676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:58:04.759789 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.743314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.5.117.3:35835)
0929 15:58:04.743530 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.743535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.743556 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:58:04.743743 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.743752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.743832 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.743934 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:04.745322 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.746110 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.747298 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.747737 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.747835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.753365 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:04.753379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:04.753397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:04.753441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:04.756722 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.757411 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.757422 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.757425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.757683 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.758134 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.758137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.758139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.758531 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:04.758781 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.759158 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.759360 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:04.760883 10169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.855746 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10047
I20260929 15:58:04.872292 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 9689
I20260929 15:58:04.873795 9750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.998234 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 9689
2026-09-29T15:58:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4865 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T15:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:05Z Disabled control of system clock
I20260929 15:58:05.088651 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37755
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:37755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:58:05.470259 10229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:05.470561 10229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:05.470708 10229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:05.477826 10229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:05.477959 10229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:05.478047 10229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:05.478137 10229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:05.478206 10229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:05.487828 10229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:37755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:05.489558 10229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:05.491796 10229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:05.505776 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:05.505805 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:05.506433 10237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:05.506886 10229 server_base.cc:1034] running on GCE node
I20260929 15:58:05.507865 10229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:05.509654 10229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:05.510921 10229 hybrid_clock.cc:648] HybridClock initialized: now 1790697485510806 us; error 110 us; skew 500 ppm
I20260929 15:58:05.511355 10229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:05.520364 10229 webserver.cc:460] Webserver started at http://127.5.117.62:40305/ using document root <none> and password file <none>
I20260929 15:58:05.521334 10229 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:05.521493 10229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:05.521903 10229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:05.525637 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "a5185c1afb1643d6b1632854d40a9e75"
format_stamp: "Formatted at 2026-09-29 15:58:05 on dist-test-slave-b69v"
server_key: "bc576f459d9a73dd9a6c21093091c451"
server_key_iv: "5479ee9e68868012304cce5e0266cb98"
server_key_version: "encryptionkey@0"
I20260929 15:58:05.526614 10229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "a5185c1afb1643d6b1632854d40a9e75"
format_stamp: "Formatted at 2026-09-29 15:58:05 on dist-test-slave-b69v"
server_key: "bc576f459d9a73dd9a6c21093091c451"
server_key_iv: "5479ee9e68868012304cce5e0266cb98"
server_key_version: "encryptionkey@0"
I20260929 15:58:05.532845 10229 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 15:58:05.537272 10243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:05.539016 10229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:05.539234 10229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "a5185c1afb1643d6b1632854d40a9e75"
format_stamp: "Formatted at 2026-09-29 15:58:05 on dist-test-slave-b69v"
server_key: "bc576f459d9a73dd9a6c21093091c451"
server_key_iv: "5479ee9e68868012304cce5e0266cb98"
server_key_version: "encryptionkey@0"
I20260929 15:58:05.539489 10229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:05.545774 10229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:05.546587 10229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:05.546902 10229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:05.567690 10229 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:37755
I20260929 15:58:05.567765 10294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:37755 every 8 connection(s)
I20260929 15:58:05.569166 10229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:05.573187 10295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:05.576741 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10229
I20260929 15:58:05.576960 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:05.577849 5588 external_mini_cluster.cc:1428] Setting key 967d456fb7b059f7b0460b231abbee7b
I20260929 15:58:05.586469 10295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: Bootstrap starting.
I20260929 15:58:05.589396 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:05.580603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.5.117.62:37755)
0929 15:58:05.581852 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:05.581865 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:05.581918 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:58:05.584849 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:05.584881 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:05.585443 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:05.585876 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:05.587338 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:05.587347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:05.587363 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:05.587383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:05.587459 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:05.587478 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:05.587698 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:05.587815 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 15:58:05.588189 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":438,"threads_started":1}
I20260929 15:58:05.590997 10295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:05.592381 10295 log.cc:824] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:05.596031 10295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: No bootstrap required, opened a new log
I20260929 15:58:05.604585 10295 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } }
I20260929 15:58:05.605041 10295 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:05.605121 10295 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5185c1afb1643d6b1632854d40a9e75, State: Initialized, Role: FOLLOWER
I20260929 15:58:05.605763 10295 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } }
I20260929 15:58:05.605997 10295 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:05.606141 10295 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:05.606314 10295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:05.609148 10295 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } }
I20260929 15:58:05.609724 10295 leader_election.cc:302] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5185c1afb1643d6b1632854d40a9e75; no voters:
I20260929 15:58:05.610633 10295 leader_election.cc:288] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:05.610741 10300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:05.611699 10300 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [term 1 LEADER]: Becoming Leader. State: Replica: a5185c1afb1643d6b1632854d40a9e75, State: Running, Role: LEADER
I20260929 15:58:05.612442 10300 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } }
I20260929 15:58:05.613440 10295 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:05.615476 10302 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5185c1afb1643d6b1632854d40a9e75. Latest consensus state: current_term: 1 leader_uuid: "a5185c1afb1643d6b1632854d40a9e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } } }
I20260929 15:58:05.615947 10302 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:05.615757 10301 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5185c1afb1643d6b1632854d40a9e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5185c1afb1643d6b1632854d40a9e75" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37755 } } }
I20260929 15:58:05.616430 10301 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:05.618647 10305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:05.624426 10305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:05.634754 10305 catalog_manager.cc:1347] Generated new cluster ID: 2942e5a81b614e0fae54b60d09f31873
I20260929 15:58:05.634899 10305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:05.652606 10305 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:05.653136 10305 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:05.663933 10305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: Generated new TSK 0
I20260929 15:58:05.664777 10305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:05.675822 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--builtin_ntp_servers=127.5.117.20:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:58:06.045872 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:06.046303 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:06.046475 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:06.058284 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:06.058440 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:06.058670 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:06.071965 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:06.074146 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:06.076820 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:06.094843 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.094756 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.094856 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:06.096213 10319 server_base.cc:1034] running on GCE node
I20260929 15:58:06.097287 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:06.099022 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:06.100370 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790697486100280 us; error 123 us; skew 500 ppm
I20260929 15:58:06.100878 10319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:06.105135 10319 webserver.cc:460] Webserver started at http://127.5.117.1:41395/ using document root <none> and password file <none>
I20260929 15:58:06.106146 10319 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:06.106273 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:06.106671 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:06.111121 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "a174b1accf774480b58163fcb3c50fb7"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "221a7989c1242f3e2dd8daacfbab8199"
server_key_iv: "d73a2ca4a2ed1fcef5a0454b4558df92"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.112555 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "a174b1accf774480b58163fcb3c50fb7"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "221a7989c1242f3e2dd8daacfbab8199"
server_key_iv: "d73a2ca4a2ed1fcef5a0454b4558df92"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.120645 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 15:58:06.125941 10333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.127907 10319 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:06.128291 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "a174b1accf774480b58163fcb3c50fb7"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "221a7989c1242f3e2dd8daacfbab8199"
server_key_iv: "d73a2ca4a2ed1fcef5a0454b4558df92"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.128629 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:06.136818 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:06.137989 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:06.138370 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:06.139890 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:06.141752 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:06.141896 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.142050 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:06.142153 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.183674 10319 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:39405
I20260929 15:58:06.183723 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:39405 every 8 connection(s)
I20260929 15:58:06.185250 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:06.191224 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10319
I20260929 15:58:06.191449 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:06.192214 5588 external_mini_cluster.cc:1428] Setting key 083053a3eb0e051407f2f086d181abb3
I20260929 15:58:06.195232 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.188812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:34495 (local address 127.5.117.62:37755)
0929 15:58:06.189288 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:06.189293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:06.190688 (+ 1395us) server_negotiation.cc:407] Connection header received
0929 15:58:06.192730 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:06.192737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:06.192824 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:06.192955 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:06.194389 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:06.194394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:06.194399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:06.194418 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:06.194473 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:06.194485 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:06.194619 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:06.194825 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 15:58:06.194901 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
I20260929 15:58:06.196192 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.189494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37755 (local address 127.5.117.1:34495)
0929 15:58:06.190523 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:06.190561 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:06.192450 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:06.193210 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:06.193218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:06.193305 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:06.194080 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:06.194083 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:06.194122 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:06.194218 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:06.194583 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:06.194594 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:06.194603 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 15:58:06.194896 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 15:58:06.195139 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":316,"threads_started":1}
I20260929 15:58:06.196259 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--builtin_ntp_servers=127.5.117.20:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:58:06.197854 10446 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37755
I20260929 15:58:06.198176 10446 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:06.198812 10446 heartbeater.cc:511] Master 127.5.117.62:37755 requested a full tablet report, sending...
I20260929 15:58:06.201231 10260 ts_manager.cc:194] Registered new tserver with Master: a174b1accf774480b58163fcb3c50fb7 (127.5.117.1:39405)
W20260929 15:58:06.600703 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:06.601109 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:06.601255 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:06.608871 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:06.608971 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:06.609095 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:06.619112 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:06.620787 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:06.622902 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:06.637485 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.637543 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.637570 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:06.638940 10450 server_base.cc:1034] running on GCE node
I20260929 15:58:06.639622 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:06.640700 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:06.641934 10450 hybrid_clock.cc:648] HybridClock initialized: now 1790697486641885 us; error 59 us; skew 500 ppm
I20260929 15:58:06.642274 10450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:06.644884 10450 webserver.cc:460] Webserver started at http://127.5.117.2:44057/ using document root <none> and password file <none>
I20260929 15:58:06.645637 10450 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:06.645774 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:06.646126 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:06.649185 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "f2f20959ef324ba2b30a0bb454b44047"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "f7d1169f06fef861f2783db82f7dcf6d"
server_key_iv: "c14e003737ef237ca24f813f899de9be"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.649945 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "f2f20959ef324ba2b30a0bb454b44047"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "f7d1169f06fef861f2783db82f7dcf6d"
server_key_iv: "c14e003737ef237ca24f813f899de9be"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.655117 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:06.658481 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.659688 10450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:58:06.659855 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "f2f20959ef324ba2b30a0bb454b44047"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-b69v"
server_key: "f7d1169f06fef861f2783db82f7dcf6d"
server_key_iv: "c14e003737ef237ca24f813f899de9be"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.660086 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:06.665834 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:06.666662 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:06.666942 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:06.667874 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:06.669272 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:06.669395 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.669513 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:06.669600 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.703502 10450 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:34467
I20260929 15:58:06.703550 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:34467 every 8 connection(s)
I20260929 15:58:06.704906 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:06.713259 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10450
I20260929 15:58:06.713454 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:06.713955 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.707749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:47781 (local address 127.5.117.62:37755)
0929 15:58:06.708294 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:06.708302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:06.709494 (+ 1192us) server_negotiation.cc:407] Connection header received
0929 15:58:06.711128 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:06.711132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:06.711188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:06.711274 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:06.712505 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:06.712510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:06.712513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:06.712528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:06.712572 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:06.712583 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:06.712715 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:06.713513 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 15:58:06.713589 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":284,"threads_started":1}
I20260929 15:58:06.714097 5588 external_mini_cluster.cc:1428] Setting key ddfb3cb52cd4d24bd85217920557e547
I20260929 15:58:06.714979 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.708390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37755 (local address 127.5.117.2:47781)
0929 15:58:06.709342 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:06.709379 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:06.710892 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:06.711462 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:06.711470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:06.711548 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:06.712230 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:06.712232 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:06.712272 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:06.712349 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:06.712665 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:06.712672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:06.712676 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 15:58:06.713591 (+ 915us) client_negotiation.cc:240] Negotiation successful
0929 15:58:06.713790 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
I20260929 15:58:06.716456 10577 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37755
I20260929 15:58:06.716766 10577 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:06.717347 10577 heartbeater.cc:511] Master 127.5.117.62:37755 requested a full tablet report, sending...
I20260929 15:58:06.717350 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--builtin_ntp_servers=127.5.117.20:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:58:06.718837 10260 ts_manager.cc:194] Registered new tserver with Master: f2f20959ef324ba2b30a0bb454b44047 (127.5.117.2:34467)
W20260929 15:58:07.063468 10581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:07.063776 10581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:07.063931 10581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:07.071108 10581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:07.071197 10581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:07.071306 10581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:07.080096 10581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:07.081403 10581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:07.083276 10581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:07.097636 10589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.097744 10586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.097703 10587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:07.098640 10581 server_base.cc:1034] running on GCE node
I20260929 15:58:07.099221 10581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:07.100240 10581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:07.101459 10581 hybrid_clock.cc:648] HybridClock initialized: now 1790697487101406 us; error 43 us; skew 500 ppm
I20260929 15:58:07.101775 10581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:07.104394 10581 webserver.cc:460] Webserver started at http://127.5.117.3:33045/ using document root <none> and password file <none>
I20260929 15:58:07.105099 10581 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:07.105248 10581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:07.105693 10581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:07.109485 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "614892d1f24142799caadfb3a3c9a087"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-b69v"
server_key: "60f3f65206195ec8246beba3e57aaff8"
server_key_iv: "9b6d1bd81549c5f01de97134555eb553"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.110255 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "614892d1f24142799caadfb3a3c9a087"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-b69v"
server_key: "60f3f65206195ec8246beba3e57aaff8"
server_key_iv: "9b6d1bd81549c5f01de97134555eb553"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.116544 10581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 15:58:07.120827 10595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.122954 10581 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:07.123169 10581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "614892d1f24142799caadfb3a3c9a087"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-b69v"
server_key: "60f3f65206195ec8246beba3e57aaff8"
server_key_iv: "9b6d1bd81549c5f01de97134555eb553"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.123454 10581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:07.130388 10581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:07.131253 10581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:07.131538 10581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:07.132601 10581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:07.134316 10581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:07.134414 10581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.134573 10581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:07.134660 10581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.174281 10581 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34541
I20260929 15:58:07.174331 10707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34541 every 8 connection(s)
I20260929 15:58:07.176010 10581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:07.179553 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10581
I20260929 15:58:07.179744 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:07.180384 5588 external_mini_cluster.cc:1428] Setting key 4ad9dc782c3374e20e41c189cf5085d2
I20260929 15:58:07.186157 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.179418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36129 (local address 127.5.117.62:37755)
0929 15:58:07.179698 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.179705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.181421 (+ 1716us) server_negotiation.cc:407] Connection header received
0929 15:58:07.183688 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.183693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.183777 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.183901 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.185345 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.185351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.185356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.185375 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.185419 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.185438 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.185574 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.185790 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.185863 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 15:58:07.187111 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.179974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37755 (local address 127.5.117.3:36129)
0929 15:58:07.181229 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.181285 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.183349 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.184141 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.184149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.184242 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.185039 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.185042 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.185081 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.185173 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.185578 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.185586 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.185592 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.185820 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.186013 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":393,"threads_started":1}
I20260929 15:58:07.188747 10708 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37755
I20260929 15:58:07.189111 10708 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:07.189875 10708 heartbeater.cc:511] Master 127.5.117.62:37755 requested a full tablet report, sending...
I20260929 15:58:07.191484 10260 ts_manager.cc:194] Registered new tserver with Master: 614892d1f24142799caadfb3a3c9a087 (127.5.117.3:34541)
I20260929 15:58:07.196154 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:58:07.204347 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.201572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.5.117.62:37755)
0929 15:58:07.201759 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.201765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.202344 (+ 579us) server_negotiation.cc:407] Connection header received
0929 15:58:07.202444 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.202448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.202500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.202599 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.203600 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.203605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.203608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.203628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.203660 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.203672 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.203787 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.203915 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.203960 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:58:07.204641 10446 heartbeater.cc:503] Master 127.5.117.62:37755 was elected leader, sending a full tablet report...
I20260929 15:58:07.208823 10260 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:07.216583 10260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:07.252235 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.230196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.5.117.2:34467)
0929 15:58:07.245804 (+ 15608us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.245821 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.246628 (+ 807us) server_negotiation.cc:407] Connection header received
0929 15:58:07.247211 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.247228 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.247669 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.247853 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.251133 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.251146 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.251166 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.251198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.251274 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.251312 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.251474 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.251662 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.251793 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15289,"thread_start_us":16182,"threads_started":1}
I20260929 15:58:07.252270 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.230003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:34467 (local address 127.0.0.1:41564)
0929 15:58:07.246121 (+ 16118us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.246206 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.246827 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.249265 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.249279 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.249427 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.250678 (+ 1251us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.250681 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.250721 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.250843 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.251432 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.251444 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.251454 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.251652 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.251752 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15657,"thread_start_us":20654,"threads_started":1}
I20260929 15:58:07.255119 10720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.247811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49034)
0929 15:58:07.251253 (+ 3442us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.251273 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.251403 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.252555 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.252558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.252586 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.253771 (+ 1185us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.253776 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.253795 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.253845 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.254391 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.254397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.254405 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.254556 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.254623 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3291,"thread_start_us":2419,"threads_started":1}
I20260929 15:58:07.255409 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.230153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34541 (local address 127.0.0.1:45774)
0929 15:58:07.251143 (+ 20990us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.251159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.251300 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.253229 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.253234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.253257 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.253973 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.253976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.253991 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.254032 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.254633 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.254636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.254639 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.254806 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.254862 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20824,"thread_start_us":16591,"threads_started":1}
I20260929 15:58:07.255720 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.230500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.5.117.3:34541)
0929 15:58:07.246497 (+ 15997us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.246507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.251321 (+ 4814us) server_negotiation.cc:407] Connection header received
0929 15:58:07.251435 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.251444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.251719 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.251888 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.254234 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.254242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.254254 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.254275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.254334 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.254356 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.254530 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.254780 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.254888 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15786,"thread_start_us":15576,"threads_started":1}
I20260929 15:58:07.256950 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.248203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.5.117.1:39405)
0929 15:58:07.249106 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.249121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.251588 (+ 2467us) server_negotiation.cc:407] Connection header received
0929 15:58:07.251692 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.251700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.251905 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.252099 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.254039 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.254047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.254061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.254079 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.254143 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.254166 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.254339 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.255474 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.255589 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":362,"threads_started":1}
I20260929 15:58:07.258260 10512 tablet_service.cc:1467] Processing CreateTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a (DEFAULT_TABLE table=test-table [id=abcd9639d4d94ac4bfb66d72f62b8d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:07.259742 10512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef10d1c418ab4c49a0063a7748fdd14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:07.259518 10643 tablet_service.cc:1467] Processing CreateTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a (DEFAULT_TABLE table=test-table [id=abcd9639d4d94ac4bfb66d72f62b8d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:07.261001 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef10d1c418ab4c49a0063a7748fdd14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:07.260541 10381 tablet_service.cc:1467] Processing CreateTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a (DEFAULT_TABLE table=test-table [id=abcd9639d4d94ac4bfb66d72f62b8d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:07.261874 10381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef10d1c418ab4c49a0063a7748fdd14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:07.273617 10726 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Bootstrap starting.
I20260929 15:58:07.276985 10727 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Bootstrap starting.
I20260929 15:58:07.278115 10726 tablet_bootstrap.cc:654] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:07.279559 10726 log.cc:824] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:07.280422 10728 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Bootstrap starting.
I20260929 15:58:07.280953 10727 tablet_bootstrap.cc:654] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:07.282120 10727 log.cc:824] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:07.284297 10728 tablet_bootstrap.cc:654] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:07.284797 10726 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: No bootstrap required, opened a new log
I20260929 15:58:07.285089 10726 ts_tablet_manager.cc:1397] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 15:58:07.285320 10727 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: No bootstrap required, opened a new log
I20260929 15:58:07.285324 10728 log.cc:824] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:07.285602 10727 ts_tablet_manager.cc:1397] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:58:07.287636 10728 tablet_bootstrap.cc:492] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: No bootstrap required, opened a new log
I20260929 15:58:07.287889 10728 ts_tablet_manager.cc:1397] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 15:58:07.295437 10728 raft_consensus.cc:348] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.295895 10728 raft_consensus.cc:374] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:07.296027 10728 raft_consensus.cc:729] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a174b1accf774480b58163fcb3c50fb7, State: Initialized, Role: FOLLOWER
I20260929 15:58:07.296835 10728 consensus_queue.cc:260] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.296862 10726 raft_consensus.cc:348] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.296861 10727 raft_consensus.cc:348] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.297307 10727 raft_consensus.cc:374] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:07.297307 10726 raft_consensus.cc:374] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:07.297420 10726 raft_consensus.cc:729] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f20959ef324ba2b30a0bb454b44047, State: Initialized, Role: FOLLOWER
I20260929 15:58:07.297420 10727 raft_consensus.cc:729] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614892d1f24142799caadfb3a3c9a087, State: Initialized, Role: FOLLOWER
I20260929 15:58:07.298111 10727 consensus_queue.cc:260] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.298275 10726 consensus_queue.cc:260] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.299736 10728 ts_tablet_manager.cc:1428] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 15:58:07.300822 10577 heartbeater.cc:503] Master 127.5.117.62:37755 was elected leader, sending a full tablet report...
I20260929 15:58:07.301374 10708 heartbeater.cc:503] Master 127.5.117.62:37755 was elected leader, sending a full tablet report...
I20260929 15:58:07.301923 10727 ts_tablet_manager.cc:1428] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260929 15:58:07.303341 10726 ts_tablet_manager.cc:1428] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260929 15:58:07.423731 10734 raft_consensus.cc:482] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:07.424268 10734 raft_consensus.cc:504] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.426673 10734 leader_election.cc:288] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 614892d1f24142799caadfb3a3c9a087 (127.5.117.3:34541), a174b1accf774480b58163fcb3c50fb7 (127.5.117.1:39405)
W20260929 15:58:07.428221 10709 tablet.cc:2369] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:07.431794 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.427443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:54899 (local address 127.5.117.3:34541)
0929 15:58:07.427694 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.427700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.428699 (+ 999us) server_negotiation.cc:407] Connection header received
0929 15:58:07.428995 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.429001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.429097 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.429189 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.430871 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.430878 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.430882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.430906 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.430952 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.430968 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.431117 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.431398 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.431477 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:58:07.431840 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.427446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34541 (local address 127.5.117.2:54899)
0929 15:58:07.428566 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.428591 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.428821 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.429477 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.429486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.429524 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.430536 (+ 1012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.430539 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.430559 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.430610 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.431107 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.431114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.431121 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.431347 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.431400 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":606,"threads_started":1}
I20260929 15:58:07.432682 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.429285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:47859 (local address 127.5.117.1:39405)
0929 15:58:07.429606 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.429613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.430115 (+ 502us) server_negotiation.cc:407] Connection header received
0929 15:58:07.430229 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.430236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.430326 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.430490 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.431761 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.431780 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.431791 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.431818 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.431871 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.431892 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.432030 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.432278 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.432373 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260929 15:58:07.432883 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.428741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.5.117.2:47859)
0929 15:58:07.429920 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.429940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.430072 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.430541 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.430546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.430569 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.431474 (+ 905us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:07.431477 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.431495 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:07.431565 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.432143 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.432153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.432157 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.432309 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.432362 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"thread_start_us":790,"threads_started":1}
I20260929 15:58:07.433185 10663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef10d1c418ab4c49a0063a7748fdd14a" candidate_uuid: "f2f20959ef324ba2b30a0bb454b44047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614892d1f24142799caadfb3a3c9a087" is_pre_election: true
I20260929 15:58:07.433777 10663 raft_consensus.cc:2454] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2f20959ef324ba2b30a0bb454b44047 in term 0.
I20260929 15:58:07.433964 10401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef10d1c418ab4c49a0063a7748fdd14a" candidate_uuid: "f2f20959ef324ba2b30a0bb454b44047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a174b1accf774480b58163fcb3c50fb7" is_pre_election: true
I20260929 15:58:07.434571 10468 leader_election.cc:302] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 614892d1f24142799caadfb3a3c9a087, f2f20959ef324ba2b30a0bb454b44047; no voters:
I20260929 15:58:07.434664 10401 raft_consensus.cc:2454] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2f20959ef324ba2b30a0bb454b44047 in term 0.
I20260929 15:58:07.435070 10734 raft_consensus.cc:2781] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:07.435213 10734 raft_consensus.cc:482] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:07.435338 10734 raft_consensus.cc:3037] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:07.437907 10734 raft_consensus.cc:504] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
W20260929 15:58:07.438119 10447 tablet.cc:2369] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:07.438544 10734 leader_election.cc:288] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [CANDIDATE]: Term 1 election: Requested vote from peers 614892d1f24142799caadfb3a3c9a087 (127.5.117.3:34541), a174b1accf774480b58163fcb3c50fb7 (127.5.117.1:39405)
I20260929 15:58:07.439021 10663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef10d1c418ab4c49a0063a7748fdd14a" candidate_uuid: "f2f20959ef324ba2b30a0bb454b44047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614892d1f24142799caadfb3a3c9a087"
I20260929 15:58:07.439249 10663 raft_consensus.cc:3037] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:07.439410 10401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef10d1c418ab4c49a0063a7748fdd14a" candidate_uuid: "f2f20959ef324ba2b30a0bb454b44047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a174b1accf774480b58163fcb3c50fb7"
I20260929 15:58:07.439638 10401 raft_consensus.cc:3037] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:07.442778 10663 raft_consensus.cc:2454] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2f20959ef324ba2b30a0bb454b44047 in term 1.
I20260929 15:58:07.443027 10401 raft_consensus.cc:2454] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2f20959ef324ba2b30a0bb454b44047 in term 1.
I20260929 15:58:07.443198 10468 leader_election.cc:302] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 614892d1f24142799caadfb3a3c9a087, f2f20959ef324ba2b30a0bb454b44047; no voters:
I20260929 15:58:07.443538 10734 raft_consensus.cc:2781] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:07.443987 10734 raft_consensus.cc:686] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 1 LEADER]: Becoming Leader. State: Replica: f2f20959ef324ba2b30a0bb454b44047, State: Running, Role: LEADER
I20260929 15:58:07.444617 10734 consensus_queue.cc:237] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } }
I20260929 15:58:07.449815 10260 catalog_manager.cc:5495] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2f20959ef324ba2b30a0bb454b44047 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "f2f20959ef324ba2b30a0bb454b44047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f20959ef324ba2b30a0bb454b44047" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 } health_report { overall_health: UNKNOWN } } }
W20260929 15:58:07.457007 10578 tablet.cc:2369] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:07.479609 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.477619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.5.117.2:34467)
0929 15:58:07.477768 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.477774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.477991 (+ 217us) server_negotiation.cc:407] Connection header received
0929 15:58:07.478179 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.478182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.478261 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.478340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.479139 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.479143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.479146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.479161 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.479193 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.479204 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.479287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.479357 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.479400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 15:58:07.487727 10663 raft_consensus.cc:1261] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 1 FOLLOWER]: Refusing update from remote peer f2f20959ef324ba2b30a0bb454b44047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:07.487730 10401 raft_consensus.cc:1261] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 1 FOLLOWER]: Refusing update from remote peer f2f20959ef324ba2b30a0bb454b44047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:07.488724 10737 consensus_queue.cc:1035] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "614892d1f24142799caadfb3a3c9a087" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:07.489065 10734 consensus_queue.cc:1035] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a174b1accf774480b58163fcb3c50fb7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:07.500905 10740 mvcc.cc:205] Tried to move back new op lower bound from 7334696908739866624 to 7334696908577673216. Current Snapshot: MvccSnapshot[applied={T|T < 7334696908739866624}]
I20260929 15:58:07.501159 10741 mvcc.cc:205] Tried to move back new op lower bound from 7334696908739866624 to 7334696908577673216. Current Snapshot: MvccSnapshot[applied={T|T < 7334696908739866624}]
I20260929 15:58:07.522819 10259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:07.523173 10259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:07.527427 10259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5185c1afb1643d6b1632854d40a9e75: Sending DeleteTablet for 3 replicas of tablet ef10d1c418ab4c49a0063a7748fdd14a
I20260929 15:58:07.529104 10512 tablet_service.cc:1514] Processing DeleteTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:07 UTC) from {username='slave'} at 127.0.0.1:41564
I20260929 15:58:07.529096 10381 tablet_service.cc:1514] Processing DeleteTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:07 UTC) from {username='slave'} at 127.0.0.1:49034
I20260929 15:58:07.530218 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10319
I20260929 15:58:07.530371 10643 tablet_service.cc:1514] Processing DeleteTablet for tablet ef10d1c418ab4c49a0063a7748fdd14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:07 UTC) from {username='slave'} at 127.0.0.1:45774
I20260929 15:58:07.541991 10751 tablet_replica.cc:330] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: stopping tablet replica
I20260929 15:58:07.542621 10751 raft_consensus.cc:2229] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:07.543921 10750 tablet_replica.cc:330] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: stopping tablet replica
I20260929 15:58:07.544503 10750 raft_consensus.cc:2229] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:07.545424 10750 raft_consensus.cc:2258] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:07.545929 10751 raft_consensus.cc:2258] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:07.547484 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.531514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.5.117.1:39405)
0929 15:58:07.531713 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.531720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.544547 (+ 12827us) server_negotiation.cc:407] Connection header received
0929 15:58:07.544811 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.544817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.544934 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.545046 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.546589 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.546595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.546601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.546624 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.546667 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.546683 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.546801 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.547069 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.547144 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:07.548635 10441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:07.549031 10752 tablet_replica.cc:330] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: stopping tablet replica
I20260929 15:58:07.549427 10752 raft_consensus.cc:2229] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:07.549655 10750 ts_tablet_manager.cc:1905] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:07.549908 10752 raft_consensus.cc:2258] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:07.553066 10752 ts_tablet_manager.cc:1905] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:07.557825 10750 ts_tablet_manager.cc:1918] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:07.558003 10750 log.cc:1196] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/ef10d1c418ab4c49a0063a7748fdd14a
I20260929 15:58:07.558651 10750 ts_tablet_manager.cc:1939] T ef10d1c418ab4c49a0063a7748fdd14a P f2f20959ef324ba2b30a0bb454b44047: Deleting consensus metadata
I20260929 15:58:07.560868 10246 catalog_manager.cc:4841] TS f2f20959ef324ba2b30a0bb454b44047 (127.5.117.2:34467): tablet ef10d1c418ab4c49a0063a7748fdd14a (table test-table [id=abcd9639d4d94ac4bfb66d72f62b8d98]) successfully deleted
I20260929 15:58:07.563120 10752 ts_tablet_manager.cc:1918] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:07.563288 10752 log.cc:1196] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/ef10d1c418ab4c49a0063a7748fdd14a
I20260929 15:58:07.563944 10752 ts_tablet_manager.cc:1939] T ef10d1c418ab4c49a0063a7748fdd14a P 614892d1f24142799caadfb3a3c9a087: Deleting consensus metadata
I20260929 15:58:07.565605 10247 catalog_manager.cc:4841] TS 614892d1f24142799caadfb3a3c9a087 (127.5.117.3:34541): tablet ef10d1c418ab4c49a0063a7748fdd14a (table test-table [id=abcd9639d4d94ac4bfb66d72f62b8d98]) successfully deleted
I20260929 15:58:07.551860 10751 ts_tablet_manager.cc:1905] T ef10d1c418ab4c49a0063a7748fdd14a P a174b1accf774480b58163fcb3c50fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:07.672695 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10319
I20260929 15:58:07.693630 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10450
W20260929 15:58:07.693581 10247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.117.1:39405 (error 108)
I20260929 15:58:07.696089 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.695516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49066)
0929 15:58:07.695691 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.695841 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 15:58:07.696382 10247 catalog_manager.cc:4584] TS a174b1accf774480b58163fcb3c50fb7 (127.5.117.1:39405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef10d1c418ab4c49a0063a7748fdd14a: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
I20260929 15:58:07.697646 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.694641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.5.117.2:34467)
0929 15:58:07.694895 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.694903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.695246 (+ 343us) server_negotiation.cc:407] Connection header received
0929 15:58:07.695512 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.695519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.695602 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.695731 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.696891 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.696898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.696904 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.696931 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.696991 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.697009 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.697145 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.697235 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.697301 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:58:07.698555 10572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:07.750433 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.749904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49068)
0929 15:58:07.750116 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.750248 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:58:07.751528 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.751190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49076)
0929 15:58:07.751342 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.751411 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 15:58:07.812022 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.811515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49084)
0929 15:58:07.811730 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.811836 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 15:58:07.813199 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.812868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49094)
0929 15:58:07.813004 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.813071 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 15:58:07.818300 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10450
I20260929 15:58:07.841426 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10581
I20260929 15:58:07.845072 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.842338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.5.117.3:34541)
0929 15:58:07.842638 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.842644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.842696 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:58:07.842774 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.842779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.842894 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.842998 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.844363 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.844367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.844371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:07.844385 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.844418 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:07.844430 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.844629 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.844688 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.844748 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:58:07.845927 10703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:07.894192 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.893409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49108)
0929 15:58:07.893771 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.893936 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260929 15:58:07.895571 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.895058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49124)
0929 15:58:07.895304 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.895391 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 15:58:07.940866 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10581
I20260929 15:58:07.959493 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10229
I20260929 15:58:07.960810 10290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:08.083570 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.082814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39405 (local address 127.0.0.1:49138)
0929 15:58:08.083136 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.083295 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 15:58:08.083843 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10229
2026-09-29T15:58:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3080 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 15:58:08.130331 5588 mini_oidc.cc:152] Starting JWKS server
I20260929 15:58:08.133906 5588 webserver.cc:460] Webserver started at https://127.0.0.1:45739/ using document root <none> and password file <none>
I20260929 15:58:08.134333 5588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:58:08.135172 5588 webserver.cc:460] Webserver started at http://127.0.0.1:36899/ using document root <none> and password file <none>
2026-09-29T15:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:08Z Disabled control of system clock
I20260929 15:58:08.175585 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:44145
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:44145
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/testchainca.cert with env {}
W20260929 15:58:08.556596 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:08.557063 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:08.557229 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:08.567045 10764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:08.567281 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:08.567433 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:08.567566 10764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:08.567651 10764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:58:08.567751 10764 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:58:08.567864 10764 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45739/jwks.json
I20260929 15:58:08.580806 10764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:44145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:08.582399 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:08.583925 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:08.598002 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:08.597993 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:08.597957 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:08.598980 10764 server_base.cc:1034] running on GCE node
I20260929 15:58:08.599826 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:08.601507 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:08.602761 10764 hybrid_clock.cc:648] HybridClock initialized: now 1790697488602720 us; error 72 us; skew 500 ppm
I20260929 15:58:08.603174 10764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:08.605917 10764 webserver.cc:460] Webserver started at http://127.5.117.62:46235/ using document root <none> and password file <none>
I20260929 15:58:08.606760 10764 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:08.606915 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:08.607281 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:08.611063 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "51599556f40b4df5a874cd785b6ea4ca"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-b69v"
server_key: "ea1865066b437fdd198a206609fc9c89"
server_key_iv: "c3cc514dd4db4d7f348639ba2e24a814"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.612027 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "51599556f40b4df5a874cd785b6ea4ca"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-b69v"
server_key: "ea1865066b437fdd198a206609fc9c89"
server_key_iv: "c3cc514dd4db4d7f348639ba2e24a814"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.619240 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260929 15:58:08.623944 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:08.625964 10764 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 15:58:08.626223 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "51599556f40b4df5a874cd785b6ea4ca"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-b69v"
server_key: "ea1865066b437fdd198a206609fc9c89"
server_key_iv: "c3cc514dd4db4d7f348639ba2e24a814"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.626531 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:08.691354 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:08.692514 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:08.692905 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:08.715267 10764 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:44145
I20260929 15:58:08.715287 10829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:44145 every 8 connection(s)
I20260929 15:58:08.716921 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:08.721004 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10764
I20260929 15:58:08.721269 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:08.721777 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:08.722047 5588 external_mini_cluster.cc:1428] Setting key c0324f2c416955f733a00a4c23d6b6a3
I20260929 15:58:08.738668 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.724296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.5.117.62:44145)
0929 15:58:08.725520 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.725535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.725593 (+ 58us) server_negotiation.cc:407] Connection header received
0929 15:58:08.727457 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.727487 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.728155 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.728628 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.729357 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.731547 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.733139 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.734168 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.734356 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.735702 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.735717 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.735734 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:08.736582 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.736682 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:08.736707 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.736819 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.737056 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.737372 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":349,"threads_started":1}
I20260929 15:58:08.738979 10830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca: Bootstrap starting.
I20260929 15:58:08.742599 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:58:08.742688 10830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:08.743767 10830 log.cc:824] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:08.746865 10830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca: No bootstrap required, opened a new log
I20260929 15:58:08.754776 10830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } }
I20260929 15:58:08.755250 10830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:08.755379 10830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51599556f40b4df5a874cd785b6ea4ca, State: Initialized, Role: FOLLOWER
I20260929 15:58:08.756161 10830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } }
I20260929 15:58:08.756436 10830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:08.756601 10830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:08.756811 10830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:08.759217 10830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } }
I20260929 15:58:08.759851 10830 leader_election.cc:302] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51599556f40b4df5a874cd785b6ea4ca; no voters:
I20260929 15:58:08.760641 10830 leader_election.cc:288] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:08.760823 10835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:08.761603 10835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [term 1 LEADER]: Becoming Leader. State: Replica: 51599556f40b4df5a874cd785b6ea4ca, State: Running, Role: LEADER
I20260929 15:58:08.762194 10835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } }
I20260929 15:58:08.763126 10830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:08.765744 10837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51599556f40b4df5a874cd785b6ea4ca. Latest consensus state: current_term: 1 leader_uuid: "51599556f40b4df5a874cd785b6ea4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } } }
I20260929 15:58:08.765750 10836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51599556f40b4df5a874cd785b6ea4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51599556f40b4df5a874cd785b6ea4ca" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44145 } } }
I20260929 15:58:08.766312 10837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:08.766322 10836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:08.767658 10842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:08.774307 10842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:08.785171 10842 catalog_manager.cc:1347] Generated new cluster ID: 96ad3f1cdf27430bbae47a64e61267e9
I20260929 15:58:08.785343 10842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:08.820523 10842 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:08.821941 10842 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:08.831271 10842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51599556f40b4df5a874cd785b6ea4ca: Generated new TSK 0
I20260929 15:58:08.832453 10842 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:58:08.874228 10860 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 15:58:08.874461 10860 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:58:08.875829 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.852807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.5.117.62:44145)
0929 15:58:08.853017 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.853024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.853611 (+ 587us) server_negotiation.cc:407] Connection header received
0929 15:58:08.853898 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.853904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.853996 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.854151 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:58:08.854671 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.855883 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.857393 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.858118 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.858267 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.858874 (+ 607us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:58:08.875003 (+ 16129us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:58:08.875452 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 15:58:08.875976 10833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 15:58:08.877584 10859 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:08.852712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:44145 (local address 127.0.0.1:57710)
0929 15:58:08.853501 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.853517 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.853672 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.854175 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.854179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.854189 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:58:08.854511 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.854520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.856010 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.856020 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.857244 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.857254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.858339 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.858345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.858486 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.858554 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:58:08.875362 (+ 16808us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:58:08.875726 (+ 364us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.117.62:44145: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":439,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.117.62:44145: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 15:58:08.880496 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10764
I20260929 15:58:08.882097 10825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:08.985427 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10764
2026-09-29T15:58:09Z chronyd exiting
I20260929 15:58:09.041388 5588 test_util.cc:175] -----------------------------------------------
I20260929 15:58:09.041571 5588 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697441705944-5588-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (916 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 15:58:09.045584 5588 mini_oidc.cc:152] Starting JWKS server
I20260929 15:58:09.047791 5588 webserver.cc:460] Webserver started at https://127.0.0.1:40153/ using document root <none> and password file <none>
I20260929 15:58:09.048038 5588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:58:09.048859 5588 webserver.cc:460] Webserver started at http://127.0.0.1:35545/ using document root <none> and password file <none>
2026-09-29T15:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:09Z Disabled control of system clock
I20260929 15:58:09.079771 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34317
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:42719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:34317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40153/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/testchainca.cert with env {}
W20260929 15:58:09.425503 10867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:09.425856 10867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:09.426050 10867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:09.433494 10867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:09.433625 10867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:09.433718 10867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:09.433830 10867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:09.433899 10867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:58:09.433991 10867 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:58:09.434073 10867 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40153/jwks.json
I20260929 15:58:09.443698 10867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:34317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40153/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:09.445159 10867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:09.446593 10867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:09.461931 10873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.462538 10872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.462679 10875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.463068 10867 server_base.cc:1034] running on GCE node
I20260929 15:58:09.464174 10867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:09.466037 10867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:09.467546 10867 hybrid_clock.cc:648] HybridClock initialized: now 1790697489467477 us; error 58 us; skew 500 ppm
I20260929 15:58:09.468037 10867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:09.471361 10867 webserver.cc:460] Webserver started at http://127.5.117.62:46321/ using document root <none> and password file <none>
I20260929 15:58:09.472314 10867 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:09.472464 10867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:09.472894 10867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:09.476789 10867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "a7f9ec84710043e2af29cdd86694a130"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-b69v"
server_key: "0a31bdf42f01ceb55d2dd8140be735eb"
server_key_iv: "5b8aefb6461d0243eb41e61f3fa5e3e8"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.477945 10867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "a7f9ec84710043e2af29cdd86694a130"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-b69v"
server_key: "0a31bdf42f01ceb55d2dd8140be735eb"
server_key_iv: "5b8aefb6461d0243eb41e61f3fa5e3e8"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.484681 10867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:58:09.489055 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.490741 10867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:09.490980 10867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "a7f9ec84710043e2af29cdd86694a130"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-b69v"
server_key: "0a31bdf42f01ceb55d2dd8140be735eb"
server_key_iv: "5b8aefb6461d0243eb41e61f3fa5e3e8"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.491267 10867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:09.519423 10867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:09.520695 10867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:09.521072 10867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:09.543689 10867 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:34317
I20260929 15:58:09.543917 10932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:34317 every 8 connection(s)
I20260929 15:58:09.545424 10867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:09.546855 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10867
I20260929 15:58:09.547086 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:09.548113 5588 external_mini_cluster.cc:1428] Setting key 201b97de052be49f7707f23e21cd1fc1
I20260929 15:58:09.551024 10933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:09.564724 10933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130: Bootstrap starting.
I20260929 15:58:09.565623 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.550345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.5.117.62:34317)
0929 15:58:09.551490 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.551501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.551548 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:58:09.553372 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.553404 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.553900 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.554246 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.554917 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.557110 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.559130 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.560403 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.560600 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.562193 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.562202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.562219 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:09.563037 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.563135 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:09.563161 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.563263 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.563755 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.564300 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":414,"threads_started":1}
I20260929 15:58:09.568722 10933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:09.569986 10933 log.cc:824] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:09.570701 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:58:09.573246 10933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130: No bootstrap required, opened a new log
I20260929 15:58:09.580691 10933 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } }
I20260929 15:58:09.581116 10933 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:09.581215 10933 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f9ec84710043e2af29cdd86694a130, State: Initialized, Role: FOLLOWER
I20260929 15:58:09.581804 10933 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } }
I20260929 15:58:09.582013 10933 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:09.582152 10933 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:09.582310 10933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:09.584502 10933 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } }
I20260929 15:58:09.585004 10933 leader_election.cc:302] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f9ec84710043e2af29cdd86694a130; no voters:
I20260929 15:58:09.585740 10933 leader_election.cc:288] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:09.585901 10938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:09.586655 10938 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [term 1 LEADER]: Becoming Leader. State: Replica: a7f9ec84710043e2af29cdd86694a130, State: Running, Role: LEADER
I20260929 15:58:09.587212 10938 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } }
I20260929 15:58:09.587832 10933 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:09.590571 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f9ec84710043e2af29cdd86694a130. Latest consensus state: current_term: 1 leader_uuid: "a7f9ec84710043e2af29cdd86694a130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } } }
I20260929 15:58:09.591080 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.590948 10939 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f9ec84710043e2af29cdd86694a130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f9ec84710043e2af29cdd86694a130" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34317 } } }
I20260929 15:58:09.591249 10939 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.593281 10947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:09.598359 10947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:09.608455 10947 catalog_manager.cc:1347] Generated new cluster ID: b02b2ebfd9d04f21bac20f379bc53bd2
I20260929 15:58:09.608579 10947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:09.663367 10947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:09.665403 10947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:09.674449 10947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f9ec84710043e2af29cdd86694a130: Generated new TSK 0
I20260929 15:58:09.675495 10947 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:58:09.710887 10963 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 15:58:09.711028 10963 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:58:09.712276 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.691201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.5.117.62:34317)
0929 15:58:09.691411 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.691419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.691962 (+ 543us) server_negotiation.cc:407] Connection header received
0929 15:58:09.692260 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.692268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.692370 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.692532 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:58:09.693071 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.694284 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.695666 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.696465 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.696595 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.697097 (+ 502us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:58:09.711543 (+ 14446us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:58:09.711898 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 15:58:09.712309 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:09.691036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34317 (local address 127.0.0.1:36718)
0929 15:58:09.691812 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.691872 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.692019 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.692536 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.692539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.692550 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:58:09.692884 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.692895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.694418 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.694422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.695528 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.695538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.696662 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.696666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.696804 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.696835 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:58:09.711754 (+ 14919us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:58:09.711925 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.117.62:34317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":481,"threads_started":1}
W20260929 15:58:09.712396 10936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 15:58:09.713629 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10867
I20260929 15:58:09.714614 10928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:09.817080 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10867
2026-09-29T15:58:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (830 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 15:58:09.875949 5588 mini_oidc.cc:152] Starting JWKS server
I20260929 15:58:09.878049 5588 webserver.cc:460] Webserver started at https://127.0.0.1:43705/ using document root <none> and password file <none>
I20260929 15:58:09.878316 5588 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:58:09.878942 5588 webserver.cc:460] Webserver started at http://127.0.0.1:37659/ using document root <none> and password file <none>
2026-09-29T15:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:09Z Disabled control of system clock
I20260929 15:58:09.912493 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37511
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:37511
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:58:10.283787 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:10.284194 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:10.284384 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:10.291728 10970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:10.291827 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:10.291867 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:10.291952 10970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:10.292043 10970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:58:10.292182 10970 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:58:10.292243 10970 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43705/jwks.json
I20260929 15:58:10.301432 10970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:37511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:10.303011 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:10.305225 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:10.318202 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:10.318163 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:10.318274 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:10.319108 10970 server_base.cc:1034] running on GCE node
I20260929 15:58:10.319886 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:10.321400 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:10.322635 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790697490322600 us; error 51 us; skew 500 ppm
I20260929 15:58:10.323005 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:10.325569 10970 webserver.cc:460] Webserver started at http://127.5.117.62:36757/ using document root <none> and password file <none>
I20260929 15:58:10.326328 10970 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:10.326469 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:10.326843 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:10.329952 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "36d2cde4e68c45b68ffb98f0a241d420"
format_stamp: "Formatted at 2026-09-29 15:58:10 on dist-test-slave-b69v"
server_key: "aff30c354e5625e335634b6b523d6002"
server_key_iv: "03f41e9533e4ca3a568a1a5ca2209003"
server_key_version: "encryptionkey@0"
I20260929 15:58:10.330778 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "36d2cde4e68c45b68ffb98f0a241d420"
format_stamp: "Formatted at 2026-09-29 15:58:10 on dist-test-slave-b69v"
server_key: "aff30c354e5625e335634b6b523d6002"
server_key_iv: "03f41e9533e4ca3a568a1a5ca2209003"
server_key_version: "encryptionkey@0"
I20260929 15:58:10.337069 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 15:58:10.341634 10984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:10.343343 10970 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:10.343560 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "36d2cde4e68c45b68ffb98f0a241d420"
format_stamp: "Formatted at 2026-09-29 15:58:10 on dist-test-slave-b69v"
server_key: "aff30c354e5625e335634b6b523d6002"
server_key_iv: "03f41e9533e4ca3a568a1a5ca2209003"
server_key_version: "encryptionkey@0"
I20260929 15:58:10.343811 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:10.375800 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:10.376704 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:10.377036 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:10.399266 10970 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:37511
I20260929 15:58:10.399236 11035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:37511 every 8 connection(s)
I20260929 15:58:10.400817 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:10.401403 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 10970
I20260929 15:58:10.401650 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:10.402572 5588 external_mini_cluster.cc:1428] Setting key 85d9261f647c0fc91f49614178174a28
I20260929 15:58:10.405972 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:10.417136 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.404934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.5.117.62:37511)
0929 15:58:10.406022 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.406034 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.406114 (+ 80us) server_negotiation.cc:407] Connection header received
0929 15:58:10.407908 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.407930 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.408495 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.408935 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.409532 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.410949 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.412376 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.413086 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.413281 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.414643 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.414650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.414666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:10.415271 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.415344 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:10.415368 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.415525 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.415747 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.416130 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":408,"threads_started":1}
I20260929 15:58:10.418428 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420: Bootstrap starting.
I20260929 15:58:10.421756 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:58:10.422444 11036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:10.423542 11036 log.cc:824] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:10.426584 11036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420: No bootstrap required, opened a new log
I20260929 15:58:10.433954 11036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } }
I20260929 15:58:10.434320 11036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:10.434442 11036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36d2cde4e68c45b68ffb98f0a241d420, State: Initialized, Role: FOLLOWER
I20260929 15:58:10.435081 11036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } }
I20260929 15:58:10.435362 11036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:10.435537 11036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:10.435766 11036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:10.437992 11036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } }
I20260929 15:58:10.438549 11036 leader_election.cc:302] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36d2cde4e68c45b68ffb98f0a241d420; no voters:
I20260929 15:58:10.439193 11036 leader_election.cc:288] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:10.439328 11041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:10.440178 11041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [term 1 LEADER]: Becoming Leader. State: Replica: 36d2cde4e68c45b68ffb98f0a241d420, State: Running, Role: LEADER
I20260929 15:58:10.440757 11041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } }
I20260929 15:58:10.441403 11036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:10.443786 11042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36d2cde4e68c45b68ffb98f0a241d420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } } }
I20260929 15:58:10.444437 11042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:10.444507 11043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36d2cde4e68c45b68ffb98f0a241d420. Latest consensus state: current_term: 1 leader_uuid: "36d2cde4e68c45b68ffb98f0a241d420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36d2cde4e68c45b68ffb98f0a241d420" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37511 } } }
I20260929 15:58:10.444988 11043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:10.455930 11050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:10.461647 11050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:10.471662 11050 catalog_manager.cc:1347] Generated new cluster ID: 2ddf74e549844fbd80ecc5907afcdb37
I20260929 15:58:10.471796 11050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:10.479768 11050 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:10.481446 11050 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:10.494961 11050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36d2cde4e68c45b68ffb98f0a241d420: Generated new TSK 0
I20260929 15:58:10.496209 11050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:10.532578 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.523815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.5.117.62:37511)
0929 15:58:10.524108 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.524116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.524547 (+ 431us) server_negotiation.cc:407] Connection header received
0929 15:58:10.524855 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.524859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.524940 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.525084 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:58:10.525648 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.526450 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.528548 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.529036 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.529140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.529751 (+ 611us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:58:10.531842 (+ 2091us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43705/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:58:10.532237 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43705/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 15:58:10.532842 11065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:10.523682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37511 (local address 127.0.0.1:35744)
0929 15:58:10.524428 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.524449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.524617 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.525114 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.525120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.525134 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:58:10.525472 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.525487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.526603 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.526609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.528396 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.528412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.529234 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.529242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.529446 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.529496 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:58:10.532078 (+ 2582us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43705/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:58:10.532354 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.117.62:37511: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43705/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":415,"threads_started":1}
W20260929 15:58:10.533197 11039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35744: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43705/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260929 15:58:10.535029 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 10970
I20260929 15:58:10.536190 11031 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 15:58:10.545542 11066 webserver.cc:533] Webserver: sslize failed:
E20260929 15:58:10.545682 11066 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:58:10.630525 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 10970
2026-09-29T15:58:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (813 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T15:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:10Z Disabled control of system clock
I20260929 15:58:10.719273 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39289
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39289
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/testchainca.cert with env {}
W20260929 15:58:11.088582 11071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:11.089015 11071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:11.089130 11071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:11.096645 11071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:11.096753 11071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:11.096794 11071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:11.096848 11071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:11.096887 11071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:58:11.096946 11071 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:58:11.096992 11071 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 15:58:11.107650 11071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:11.109436 11071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:11.110955 11071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:11.126332 11076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:11.126760 11077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:11.126777 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:11.128360 11071 server_base.cc:1034] running on GCE node
I20260929 15:58:11.129235 11071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:11.130815 11071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:11.132099 11071 hybrid_clock.cc:648] HybridClock initialized: now 1790697491131977 us; error 82 us; skew 500 ppm
I20260929 15:58:11.132520 11071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:11.142007 11071 webserver.cc:460] Webserver started at http://127.5.117.62:38343/ using document root <none> and password file <none>
I20260929 15:58:11.142804 11071 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:11.142928 11071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:11.143297 11071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:11.146561 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "0a2af818f6284883baac53b54a23f6db"
format_stamp: "Formatted at 2026-09-29 15:58:11 on dist-test-slave-b69v"
server_key: "011050e423673c3412e4b74dd8aa5d3d"
server_key_iv: "93a2e1d247d2a002db8d60740d0e7ac4"
server_key_version: "encryptionkey@0"
I20260929 15:58:11.147442 11071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "0a2af818f6284883baac53b54a23f6db"
format_stamp: "Formatted at 2026-09-29 15:58:11 on dist-test-slave-b69v"
server_key: "011050e423673c3412e4b74dd8aa5d3d"
server_key_iv: "93a2e1d247d2a002db8d60740d0e7ac4"
server_key_version: "encryptionkey@0"
I20260929 15:58:11.152951 11071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:58:11.157104 11085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:11.158743 11071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:11.158968 11071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "0a2af818f6284883baac53b54a23f6db"
format_stamp: "Formatted at 2026-09-29 15:58:11 on dist-test-slave-b69v"
server_key: "011050e423673c3412e4b74dd8aa5d3d"
server_key_iv: "93a2e1d247d2a002db8d60740d0e7ac4"
server_key_version: "encryptionkey@0"
I20260929 15:58:11.159210 11071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:11.180631 11071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:11.181591 11071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:11.181911 11071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:11.203831 11071 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39289
I20260929 15:58:11.203938 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39289 every 8 connection(s)
I20260929 15:58:11.205713 11071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:11.210238 11137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:11.211408 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11071
I20260929 15:58:11.211649 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:11.212636 5588 external_mini_cluster.cc:1428] Setting key 2b3a7ace094d161e38ce9d67f2807717
I20260929 15:58:11.227185 11137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db: Bootstrap starting.
I20260929 15:58:11.229362 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.214819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.5.117.62:39289)
0929 15:58:11.215971 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.215982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.216033 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:58:11.217806 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.217828 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.218310 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.218659 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.219696 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.221386 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.223461 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.224398 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.224563 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.226162 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.226171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.226191 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:11.226890 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.226986 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:11.227012 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.227102 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.227495 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.228014 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":421,"threads_started":1}
I20260929 15:58:11.231482 11137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:11.232836 11137 log.cc:824] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:11.233568 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:58:11.235821 11137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db: No bootstrap required, opened a new log
I20260929 15:58:11.243659 11137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } }
I20260929 15:58:11.244004 11137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:11.244141 11137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2af818f6284883baac53b54a23f6db, State: Initialized, Role: FOLLOWER
I20260929 15:58:11.244733 11137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } }
I20260929 15:58:11.244943 11137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:11.245046 11137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:11.245185 11137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:11.247602 11137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } }
I20260929 15:58:11.248138 11137 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2af818f6284883baac53b54a23f6db; no voters:
I20260929 15:58:11.248705 11137 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:11.248865 11142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:11.249627 11142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [term 1 LEADER]: Becoming Leader. State: Replica: 0a2af818f6284883baac53b54a23f6db, State: Running, Role: LEADER
I20260929 15:58:11.250200 11142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } }
I20260929 15:58:11.250972 11137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:11.253238 11144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2af818f6284883baac53b54a23f6db. Latest consensus state: current_term: 1 leader_uuid: "0a2af818f6284883baac53b54a23f6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } } }
I20260929 15:58:11.253716 11144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:11.257197 11143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2af818f6284883baac53b54a23f6db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2af818f6284883baac53b54a23f6db" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39289 } } }
I20260929 15:58:11.257503 11143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:11.258008 11148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:11.265493 11148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:11.276372 11148 catalog_manager.cc:1347] Generated new cluster ID: c5ba986433c34f14b4a12684176c581a
I20260929 15:58:11.276549 11148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:11.293095 11148 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:11.295023 11148 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:11.313297 11148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2af818f6284883baac53b54a23f6db: Generated new TSK 0
I20260929 15:58:11.314520 11148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:11.777824 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.338764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.5.117.62:39289)
0929 15:58:11.339015 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.339022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.339409 (+ 387us) server_negotiation.cc:407] Connection header received
0929 15:58:11.339698 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.339705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.339830 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.340104 (+ 274us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:58:11.340769 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.341986 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.343554 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.344318 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.344449 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.345013 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:58:11.777110 (+432097us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:58:11.777534 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37346: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 15:58:11.777998 11140 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37346: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 15:58:11.778321 11166 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:11.338544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39289 (local address 127.0.0.1:37346)
0929 15:58:11.339281 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.339298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.339473 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.340193 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.340197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.340210 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:58:11.340581 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.340591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.342195 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.342200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.343392 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.343403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.344495 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.344499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.344630 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.344671 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:58:11.777505 (+432834us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:58:11.777843 (+ 338us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.117.62:39289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":401,"threads_started":1}
I20260929 15:58:11.780203 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11071
I20260929 15:58:11.781428 11132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:11.901650 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11071
2026-09-29T15:58:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1256 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (489 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (478 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:12 dist-test-slave-b69v krb5kdc[11181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:12 dist-test-slave-b69v krb5kdc[11181](info): set up 2 sockets
Sep 29 15:58:12 dist-test-slave-b69v krb5kdc[11181](info): commencing operation
krb5kdc: starting...
W20260929 15:58:13.985526 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:14 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:58:14.232154 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37393
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:37393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:14.592959 11194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:14.593374 11194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:14.593571 11194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:14.601263 11194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:14.601402 11194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:14.601471 11194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:14.601572 11194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:14.601643 11194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:14.612216 11194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:37393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:14.613934 11194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:14.615509 11194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:14.630192 11200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:14.631016 11202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:14.631198 11199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:14.632165 11194 server_base.cc:1034] running on GCE node
I20260929 15:58:14.633085 11194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:14.634852 11194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:14.636124 11194 hybrid_clock.cc:648] HybridClock initialized: now 1790697494635997 us; error 55 us; skew 500 ppm
I20260929 15:58:14.636545 11194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:14 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697494, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:14.640137 11194 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:58:14.641759 11194 webserver.cc:460] Webserver started at http://127.5.117.62:38241/ using document root <none> and password file <none>
I20260929 15:58:14.642530 11194 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:14.642642 11194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:14.642941 11194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:14.646689 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "12dc5fee170645de9c6d8c3dd938694e"
format_stamp: "Formatted at 2026-09-29 15:58:14 on dist-test-slave-b69v"
server_key: "0376c9c7654f0b1e128c902539b8572c"
server_key_iv: "3c39782b3e9a7fdfa08db9d7d517bb5f"
server_key_version: "encryptionkey@0"
I20260929 15:58:14.647643 11194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "12dc5fee170645de9c6d8c3dd938694e"
format_stamp: "Formatted at 2026-09-29 15:58:14 on dist-test-slave-b69v"
server_key: "0376c9c7654f0b1e128c902539b8572c"
server_key_iv: "3c39782b3e9a7fdfa08db9d7d517bb5f"
server_key_version: "encryptionkey@0"
I20260929 15:58:14.653545 11194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 15:58:14.657349 11209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:14.658895 11194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:14.659117 11194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "12dc5fee170645de9c6d8c3dd938694e"
format_stamp: "Formatted at 2026-09-29 15:58:14 on dist-test-slave-b69v"
server_key: "0376c9c7654f0b1e128c902539b8572c"
server_key_iv: "3c39782b3e9a7fdfa08db9d7d517bb5f"
server_key_version: "encryptionkey@0"
I20260929 15:58:14.659356 11194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:14.677256 11194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:14.681810 11194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:14.682144 11194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:14.702958 11194 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:37393
I20260929 15:58:14.702983 11260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:37393 every 8 connection(s)
I20260929 15:58:14.704684 11194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:14.707005 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11194
I20260929 15:58:14.707281 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:14.708220 5588 external_mini_cluster.cc:1428] Setting key 295ce3ed4f65213438a6ba0f13927d06
I20260929 15:58:14.710220 11261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:58:14 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:14.725421 11261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e: Bootstrap starting.
I20260929 15:58:14.730275 11261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:14.731822 11261 log.cc:824] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:14.734743 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:14.710593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.5.117.62:37393)
0929 15:58:14.711393 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:14.711409 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:14.711482 (+ 73us) server_negotiation.cc:407] Connection header received
0929 15:58:14.713824 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:14.713853 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:14.714420 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:14.714920 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:14.716455 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:14.718656 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:14.719883 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:14.720943 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:14.721140 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:14.725672 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:14.725683 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:14.725704 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:14.725756 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:14.729897 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:14.730625 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:14.730641 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:14.730652 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:14.730899 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:14.731243 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:14.731248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:14.731253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:14.732036 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:14.732401 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:14.732836 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 15:58:14.733282 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":332,"threads_started":1}
I20260929 15:58:14.735886 11261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e: No bootstrap required, opened a new log
I20260929 15:58:14.744132 11261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } }
I20260929 15:58:14.744486 11261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:14.744570 11261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dc5fee170645de9c6d8c3dd938694e, State: Initialized, Role: FOLLOWER
I20260929 15:58:14.745167 11261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } }
I20260929 15:58:14.745378 11261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:14.745477 11261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:14.745648 11261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:14.748229 11261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } }
I20260929 15:58:14.748736 11261 leader_election.cc:302] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12dc5fee170645de9c6d8c3dd938694e; no voters:
I20260929 15:58:14.749360 11261 leader_election.cc:288] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:14.749532 11266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:14.750258 11266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [term 1 LEADER]: Becoming Leader. State: Replica: 12dc5fee170645de9c6d8c3dd938694e, State: Running, Role: LEADER
I20260929 15:58:14.750783 11266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } }
I20260929 15:58:14.751878 11261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:14.754371 11268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12dc5fee170645de9c6d8c3dd938694e. Latest consensus state: current_term: 1 leader_uuid: "12dc5fee170645de9c6d8c3dd938694e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } } }
I20260929 15:58:14.754964 11268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:14.755070 11267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12dc5fee170645de9c6d8c3dd938694e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dc5fee170645de9c6d8c3dd938694e" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37393 } } }
I20260929 15:58:14.755620 11267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:14.761823 11275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:14.767208 11275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:14.776774 11275 catalog_manager.cc:1347] Generated new cluster ID: d63c31dbc4254591a7e33918c7d715c0
I20260929 15:58:14.776897 11275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:14.789366 11275 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:14.791486 11275 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:14.821123 11275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12dc5fee170645de9c6d8c3dd938694e: Generated new TSK 0
I20260929 15:58:14.822574 11275 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:58:14.881709 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:15.255988 11287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:15.256539 11287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:15.256717 11287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:15.265240 11287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:15.265368 11287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:15.265498 11287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:15.276332 11287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:15.278021 11287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:15.279533 11287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:15.295121 11293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.295017 11295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.295058 11292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:15.296288 11287 server_base.cc:1034] running on GCE node
I20260929 15:58:15.297050 11287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:15.298166 11287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:15.299417 11287 hybrid_clock.cc:648] HybridClock initialized: now 1790697495299368 us; error 44 us; skew 500 ppm
I20260929 15:58:15.299808 11287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:15 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:15.303632 11287 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:58:15.305683 11287 webserver.cc:460] Webserver started at http://127.5.117.1:40011/ using document root <none> and password file <none>
I20260929 15:58:15.306569 11287 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:15.306748 11287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:15.307132 11287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:15.310782 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "d1ce94b654604f44b0e490d0f9080dff"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "d7dfee69d7864da92a18eb1e465c2441"
server_key_iv: "a07b3543fd4add650d22eb1d5a98ca03"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.311726 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "d1ce94b654604f44b0e490d0f9080dff"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "d7dfee69d7864da92a18eb1e465c2441"
server_key_iv: "a07b3543fd4add650d22eb1d5a98ca03"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.318074 11287 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 15:58:15.322402 11302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.323941 11287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:15.324231 11287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "d1ce94b654604f44b0e490d0f9080dff"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "d7dfee69d7864da92a18eb1e465c2441"
server_key_iv: "a07b3543fd4add650d22eb1d5a98ca03"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.324496 11287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:15.347100 11287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:15.351785 11287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:15.352210 11287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:15.353577 11287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:15.355219 11287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:15.355336 11287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.355501 11287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:15.355581 11287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.393452 11287 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:33039
I20260929 15:58:15.393512 11414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:33039 every 8 connection(s)
I20260929 15:58:15.395452 11287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:15.397502 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11287
I20260929 15:58:15.397753 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:15.398846 5588 external_mini_cluster.cc:1428] Setting key fdf5c443fdac67830032c1346c760e6b
Sep 29 15:58:15 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:15.418705 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:15.399238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:44859 (local address 127.5.117.62:37393)
0929 15:58:15.400142 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:15.400150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:15.401231 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 15:58:15.403394 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:15.403401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:15.403507 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:15.403646 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:15.405420 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.406225 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.407862 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.408338 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.408481 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:15.412690 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:15.412695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:15.412699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:15.412735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:15.415470 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:15.416268 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:15.416272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:15.416274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:15.416410 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:15.416862 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:15.416866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:15.416868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:15.417137 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:15.417316 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:15.418050 (+ 734us) server_negotiation.cc:299] Negotiation successful
0929 15:58:15.418308 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":613,"threads_started":1}
I20260929 15:58:15.419757 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:15.400012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37393 (local address 127.5.117.1:44859)
0929 15:58:15.401045 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:15.401104 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:15.403080 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:15.403926 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:15.403936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:15.404580 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:15.405235 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:15.405253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.406393 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.406399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:15.407703 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:15.407714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.408594 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.408605 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:15.408858 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:15.409645 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:15.409674 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:15.412465 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:15.415624 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:15.415632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:15.415647 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:15.416120 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:15.416525 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:15.416530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:15.416534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:15.416731 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:15.417273 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:15.417281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:15.417625 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 15:58:15.417931 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 15:58:15.418453 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":420,"threads_started":1}
I20260929 15:58:15.421697 11415 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37393
I20260929 15:58:15.422076 11415 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 15:58:15.422916 11415 heartbeater.cc:511] Master 127.5.117.62:37393 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
I20260929 15:58:15.426381 11226 ts_manager.cc:194] Registered new tserver with Master: d1ce94b654604f44b0e490d0f9080dff (127.5.117.1:33039)
I20260929 15:58:15.428165 11226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:44859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:58:15.455787 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:15.817094 11421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:15.817416 11421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:15.817519 11421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:15.825274 11421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:15.825479 11421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:15.825673 11421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:15.837790 11421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:15.839443 11421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:15.841039 11421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:15.857057 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.857077 11427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.857110 11429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:15.857723 11421 server_base.cc:1034] running on GCE node
I20260929 15:58:15.858505 11421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:15.859655 11421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:15.860900 11421 hybrid_clock.cc:648] HybridClock initialized: now 1790697495860851 us; error 64 us; skew 500 ppm
I20260929 15:58:15.861259 11421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:15 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:15.864936 11421 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:58:15.866439 11421 webserver.cc:460] Webserver started at http://127.5.117.2:38343/ using document root <none> and password file <none>
I20260929 15:58:15.867223 11421 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:15.867362 11421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:15.867718 11421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:15.871098 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "3f8838234d4b4ad0a83b37ed3c4a7fad"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "243c6e437f1e6830b27a9a5b69afd276"
server_key_iv: "4f1fac6ee455b1dd299535795a5b1179"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.871951 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "3f8838234d4b4ad0a83b37ed3c4a7fad"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "243c6e437f1e6830b27a9a5b69afd276"
server_key_iv: "4f1fac6ee455b1dd299535795a5b1179"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.877779 11421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:58:15.881534 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.883575 11421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:15.883852 11421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "3f8838234d4b4ad0a83b37ed3c4a7fad"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-b69v"
server_key: "243c6e437f1e6830b27a9a5b69afd276"
server_key_iv: "4f1fac6ee455b1dd299535795a5b1179"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.884133 11421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:15.898887 11421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:15.903072 11421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:15.903378 11421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:15.904413 11421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:15.905790 11421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:15.905908 11421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.906064 11421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:15.906141 11421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.946491 11421 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44987
I20260929 15:58:15.946695 11548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44987 every 8 connection(s)
I20260929 15:58:15.948041 11421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:15.951989 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11421
I20260929 15:58:15.952219 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:15.952891 5588 external_mini_cluster.cc:1428] Setting key 0e1644695534421a9850b0714385f85c
Sep 29 15:58:15 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:15.976320 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:15.951668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36751 (local address 127.5.117.62:37393)
0929 15:58:15.952273 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:15.952280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:15.953522 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 15:58:15.955894 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:15.955898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:15.955971 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:15.956140 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:15.958578 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.959384 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.961987 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.962525 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.962669 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:15.968393 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:15.968418 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:15.968428 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:15.968488 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:15.972586 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:15.973427 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:15.973436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:15.973442 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:15.973622 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:15.974076 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:15.974084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:15.974090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:15.974507 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:15.974670 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:15.975440 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 15:58:15.975720 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
I20260929 15:58:15.977044 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:15.952319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37393 (local address 127.5.117.2:36751)
0929 15:58:15.953323 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:15.953377 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:15.955558 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:15.956413 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:15.956425 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:15.957364 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:15.958338 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:15.958367 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.959569 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.959577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:15.961824 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:15.961841 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:15.962772 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:15.962782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:15.963152 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:15.964178 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:15.964214 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:15.967892 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:15.972742 (+ 4850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:15.972749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:15.972763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:15.973274 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:15.973741 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:15.973745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:15.973747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:15.973942 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:15.974646 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:15.974653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:15.975006 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 15:58:15.975256 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 15:58:15.975675 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":407,"threads_started":1}
I20260929 15:58:15.978814 11549 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37393
I20260929 15:58:15.979230 11549 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:15.980419 11549 heartbeater.cc:511] Master 127.5.117.62:37393 requested a full tablet report, sending...
I20260929 15:58:15.982376 11226 ts_manager.cc:194] Registered new tserver with Master: 3f8838234d4b4ad0a83b37ed3c4a7fad (127.5.117.2:44987)
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 15:58:15.983400 11226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:36751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:58:16.015827 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:16.395002 11555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:16.395349 11555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:16.395527 11555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:16.402681 11555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:16.402829 11555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:16.402992 11555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:16.412097 11555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:16.413564 11555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:16.415064 11555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:16.429791 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:16.430100 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:16.431286 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:16.432231 11555 server_base.cc:1034] running on GCE node
I20260929 15:58:16.432229 11415 heartbeater.cc:503] Master 127.5.117.62:37393 was elected leader, sending a full tablet report...
I20260929 15:58:16.432889 11555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:16.434904 11555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:16.436158 11555 hybrid_clock.cc:648] HybridClock initialized: now 1790697496436066 us; error 59 us; skew 500 ppm
I20260929 15:58:16.436568 11555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.444326 11555 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:58:16.445981 11555 webserver.cc:460] Webserver started at http://127.5.117.3:41729/ using document root <none> and password file <none>
I20260929 15:58:16.446792 11555 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:16.446933 11555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:16.447307 11555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:16.450934 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "3ccd11b3025549f3bd46d7c0ea3a69a2"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-b69v"
server_key: "b330b8bff40a860290b24c03237065b8"
server_key_iv: "4b147a95a55c25456487d5a900643085"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.451835 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "3ccd11b3025549f3bd46d7c0ea3a69a2"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-b69v"
server_key: "b330b8bff40a860290b24c03237065b8"
server_key_iv: "4b147a95a55c25456487d5a900643085"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.459204 11555 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 15:58:16.464847 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:16.466583 11555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 15:58:16.466763 11555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "3ccd11b3025549f3bd46d7c0ea3a69a2"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-b69v"
server_key: "b330b8bff40a860290b24c03237065b8"
server_key_iv: "4b147a95a55c25456487d5a900643085"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.466971 11555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:16.496443 11555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:16.502033 11555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:16.502321 11555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:16.503355 11555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:16.504899 11555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:16.504979 11555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:16.505077 11555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:16.505125 11555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:16.548194 11555 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:45299
I20260929 15:58:16.548296 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:45299 every 8 connection(s)
I20260929 15:58:16.550191 11555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:16.555341 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11555
I20260929 15:58:16.555539 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:16.556203 5588 external_mini_cluster.cc:1428] Setting key 991a9295de20ac28ba986629095a4f92
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:16.579775 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.553876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:48235 (local address 127.5.117.62:37393)
0929 15:58:16.554497 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.554505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.555607 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 15:58:16.558726 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.558732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.558857 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.558996 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.561521 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.562711 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.565339 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.565934 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.566084 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.571161 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.571170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.571178 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.571225 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.575579 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.576699 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.576706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.576714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.576900 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.577462 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.577467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.577470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.577905 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.578065 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.579090 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.579354 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":297,"threads_started":1}
I20260929 15:58:16.582109 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.554390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37393 (local address 127.5.117.3:48235)
0929 15:58:16.555362 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:16.555433 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:16.558382 (+ 2949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:16.559293 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:16.559311 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:16.560297 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:16.561307 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.561340 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.562885 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.562891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.565184 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.565199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.566191 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.566210 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.566537 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.567474 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:16.567510 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:16.570904 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:16.575755 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.575767 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.575787 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.576546 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.577027 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.577031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.577037 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.577333 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.578067 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:16.578078 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:16.578590 (+ 512us) client_negotiation.cc:769] Sending connection context
0929 15:58:16.578956 (+ 366us) client_negotiation.cc:240] Negotiation successful
0929 15:58:16.579995 (+ 1039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":1650,"threads_started":1}
I20260929 15:58:16.584539 11683 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37393
I20260929 15:58:16.585096 11683 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:16.586261 11683 heartbeater.cc:511] Master 127.5.117.62:37393 requested a full tablet report, sending...
I20260929 15:58:16.588395 11225 ts_manager.cc:194] Registered new tserver with Master: 3ccd11b3025549f3bd46d7c0ea3a69a2 (127.5.117.3:45299)
I20260929 15:58:16.590126 11225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:48235
I20260929 15:58:16.599617 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/test-admin.keytab.
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.635347 5588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 15:58:16.641631 11691 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:16.638592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.5.117.1:33039)
0929 15:58:16.639411 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.639419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.639449 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:58:16.639524 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.639531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.639798 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.640016 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.641059 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59396: BlockingRecv error: recv got EOF from 127.0.0.1:59396 (error 108)
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":391,"threads_started":1}
W20260929 15:58:16.641690 5588 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.643901 5588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.645450 5588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:58:16.662745 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.647358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.5.117.1:33039)
0929 15:58:16.647728 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.647744 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.648035 (+ 291us) server_negotiation.cc:407] Connection header received
0929 15:58:16.648311 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.648321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.648379 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.648487 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.649518 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.650535 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.651792 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.652551 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.652703 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.656495 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.656506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.656522 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.656555 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.659520 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.660227 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.660234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.660237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.660374 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.661105 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.661109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.661111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.661482 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.661760 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.662174 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.662422 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
W20260929 15:58:16.669884 11691 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:16.667910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.5.117.1:33039)
0929 15:58:16.668172 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.668178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.668400 (+ 222us) server_negotiation.cc:407] Connection header received
0929 15:58:16.668497 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.668501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.668559 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.668639 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.669620 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59414: BlockingRecv error: recv got EOF from 127.0.0.1:59414 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 15:58:16.670071 5588 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.671708 5588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.673254 5588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:58:16.687381 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.674550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.5.117.1:33039)
0929 15:58:16.674743 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.674750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.674966 (+ 216us) server_negotiation.cc:407] Connection header received
0929 15:58:16.675160 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.675163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.675217 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.675317 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.676263 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.676993 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.678221 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.678668 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.678797 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.682436 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.682441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.682445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.682483 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.684950 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.685562 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.685569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.685573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.685698 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.686077 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.686083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.686086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.686355 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.686641 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.686902 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.687103 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 15:58:16.691867 11691 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:16.690000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.5.117.1:33039)
0929 15:58:16.690172 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.690178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.690481 (+ 303us) server_negotiation.cc:407] Connection header received
0929 15:58:16.690655 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.690660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.690737 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.690837 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.691626 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59434: BlockingRecv error: recv got EOF from 127.0.0.1:59434 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 15:58:16.692344 5588 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697441705944-5588-0/krb5kdc/krb5cc)
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.693893 5588 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.695158 5588 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:58:16.709610 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.696345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.5.117.1:33039)
0929 15:58:16.696603 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.696609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.696767 (+ 158us) server_negotiation.cc:407] Connection header received
0929 15:58:16.696875 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.696878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.696929 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.697005 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.697931 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.698681 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.699880 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.700380 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.700476 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.704179 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.704184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.704187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.704216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.706904 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.707664 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.707668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.707670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.707819 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.708253 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.708256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.708258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.708503 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.708738 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.709047 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.709211 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:16.711236 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11287
I20260929 15:58:16.722451 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.711979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.5.117.1:33039)
0929 15:58:16.712242 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.712249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.712432 (+ 183us) server_negotiation.cc:407] Connection header received
0929 15:58:16.712526 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.712529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.712585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.712659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.713524 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.714181 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.715297 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.715691 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.715779 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.717629 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.717634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.717637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.717664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.720298 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.720845 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.720848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.720850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.720972 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.721258 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.721261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.721263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.721500 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.721723 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.721926 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.722081 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:58:16.723107 11410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:16.817391 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11287
I20260929 15:58:16.837438 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11421
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:58:16.856278 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.838521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.5.117.2:44987)
0929 15:58:16.839210 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.839239 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.839279 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:58:16.839372 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.839379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.839654 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.839808 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.840982 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.842077 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.843436 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.844259 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.844380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.849305 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.849318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.849336 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.849373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.852905 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.853718 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.853726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.853728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.853864 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.854273 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.854279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.854283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.854683 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.855041 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.855403 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.855713 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":316,"threads_started":1}
I20260929 15:58:16.857904 11544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:16.949776 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11421
I20260929 15:58:16.965493 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11555
Sep 29 15:58:16 dist-test-slave-b69v krb5kdc[11181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:58:16.983510 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.966882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.5.117.3:45299)
0929 15:58:16.967558 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.967567 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.967606 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:58:16.967680 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.967687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.967936 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.968239 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.969435 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.970549 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.972034 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.972904 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.973090 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.977511 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.977523 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.977541 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.977576 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.980617 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.981309 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.981317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.981319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.981440 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.981854 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.981865 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.981867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.982233 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.982573 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.982874 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.983154 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260929 15:58:16.985133 11678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:17.067634 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11555
I20260929 15:58:17.084331 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11194
I20260929 15:58:17.085526 11256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:17.197849 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11194
2026-09-29T15:58:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4329 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:17 dist-test-slave-b69v krb5kdc[11711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:17 dist-test-slave-b69v krb5kdc[11711](info): set up 2 sockets
Sep 29 15:58:17 dist-test-slave-b69v krb5kdc[11711](info): commencing operation
krb5kdc: starting...
W20260929 15:58:18.257302 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:18 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab.
Entry for principal oryx/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab.
I20260929 15:58:18.517237 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39627
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab
--principal=oryx/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:18.889205 11724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:18.889622 11724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:18.889814 11724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:18.897197 11724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:18.897356 11724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:18.897472 11724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:18.897581 11724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:18.897652 11724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:18.907612 11724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab
--principal=oryx/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:18.909202 11724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:18.910727 11724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:18.923818 11729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:18.923951 11732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:18.923988 11730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:18.924680 11724 server_base.cc:1034] running on GCE node
I20260929 15:58:18.925534 11724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:18.926983 11724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:18.928498 11724 hybrid_clock.cc:648] HybridClock initialized: now 1790697498928417 us; error 84 us; skew 500 ppm
I20260929 15:58:18.928879 11724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:18 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:18.932655 11724 init.cc:377] Logged in from keytab as oryx/127.5.117.62@KRBTEST.COM (short username oryx)
I20260929 15:58:18.934223 11724 webserver.cc:460] Webserver started at http://127.5.117.62:35291/ using document root <none> and password file <none>
I20260929 15:58:18.934906 11724 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:18.935010 11724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:18.935292 11724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:18.938469 11724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "ff9b4e17af224cc69aca06fe48edd8b1"
format_stamp: "Formatted at 2026-09-29 15:58:18 on dist-test-slave-b69v"
server_key: "d8b841e0d437754376498bee490f99dd"
server_key_iv: "b7c7a90504da8e85b70d5d3af1eefcc4"
server_key_version: "encryptionkey@0"
I20260929 15:58:18.939224 11724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "ff9b4e17af224cc69aca06fe48edd8b1"
format_stamp: "Formatted at 2026-09-29 15:58:18 on dist-test-slave-b69v"
server_key: "d8b841e0d437754376498bee490f99dd"
server_key_iv: "b7c7a90504da8e85b70d5d3af1eefcc4"
server_key_version: "encryptionkey@0"
I20260929 15:58:18.944810 11724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 15:58:18.948304 11739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:18.949658 11724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:18.949844 11724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "ff9b4e17af224cc69aca06fe48edd8b1"
format_stamp: "Formatted at 2026-09-29 15:58:18 on dist-test-slave-b69v"
server_key: "d8b841e0d437754376498bee490f99dd"
server_key_iv: "b7c7a90504da8e85b70d5d3af1eefcc4"
server_key_version: "encryptionkey@0"
I20260929 15:58:18.950040 11724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:18.966351 11724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:18.970451 11724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:18.970755 11724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:18.990999 11724 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39627
I20260929 15:58:18.991045 11790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39627 every 8 connection(s)
I20260929 15:58:18.992609 11724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:18.997038 11791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.000757 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11724
I20260929 15:58:19.001014 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:19.001932 5588 external_mini_cluster.cc:1428] Setting key f2926bcafe1d5f695c63a1c46325b3f7
I20260929 15:58:19.011857 11791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Bootstrap starting.
I20260929 15:58:19.016395 11791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.017758 11791 log.cc:824] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:58:19 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:19.021783 11791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: No bootstrap required, opened a new log
I20260929 15:58:19.028301 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.004371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.5.117.62:39627)
0929 15:58:19.005223 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.005236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.005283 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:58:19.007080 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.007109 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.007555 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.007913 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.009870 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.011207 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.012984 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.013669 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.013808 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.019135 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:19.019142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:19.019190 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:19.019236 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:19.023479 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.024246 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.024255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.024263 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.024483 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.024834 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.024839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.024842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.025634 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:19.025988 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.026306 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.026760 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":397,"threads_started":1}
I20260929 15:58:19.032627 11791 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } }
I20260929 15:58:19.032979 11791 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.033072 11791 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9b4e17af224cc69aca06fe48edd8b1, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.033576 11791 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } }
I20260929 15:58:19.033794 11791 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:19.033898 11791 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:19.034030 11791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.036257 11791 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } }
I20260929 15:58:19.036742 11791 leader_election.cc:302] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff9b4e17af224cc69aca06fe48edd8b1; no voters:
I20260929 15:58:19.037431 11791 leader_election.cc:288] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:19.037556 11796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:19.038650 11796 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 LEADER]: Becoming Leader. State: Replica: ff9b4e17af224cc69aca06fe48edd8b1, State: Running, Role: LEADER
I20260929 15:58:19.039165 11796 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } }
I20260929 15:58:19.039747 11791 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:19.042197 11797 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } }
I20260929 15:58:19.042639 11797 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:19.048233 11798 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9b4e17af224cc69aca06fe48edd8b1. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } }
I20260929 15:58:19.048593 11798 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:19.048899 11805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:19.055280 11805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:19.064940 11805 catalog_manager.cc:1347] Generated new cluster ID: c2d993f77c1649f4894ceb5b6d42d84c
I20260929 15:58:19.065068 11805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:19.073926 11805 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:19.075363 11805 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:19.082988 11805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Generated new TSK 0
I20260929 15:58:19.084002 11805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
I20260929 15:58:19.151733 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:19.539702 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:19.540154 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:19.540334 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:19.548205 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:19.548372 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:19.548583 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:19.559321 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:19.560896 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:19.562314 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:19.575374 11822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:19.576331 11823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:19.576807 11825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:19.577842 11817 server_base.cc:1034] running on GCE node
I20260929 15:58:19.578430 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:19.579376 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:19.580564 11817 hybrid_clock.cc:648] HybridClock initialized: now 1790697499580516 us; error 46 us; skew 500 ppm
I20260929 15:58:19.580838 11817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:19 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697499, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:19.584268 11817 init.cc:377] Logged in from keytab as oryx/127.5.117.1@KRBTEST.COM (short username oryx)
I20260929 15:58:19.585868 11817 webserver.cc:460] Webserver started at http://127.5.117.1:37003/ using document root <none> and password file <none>
I20260929 15:58:19.586611 11817 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:19.586711 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:19.586980 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:19.590235 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "575c717d52484db0950744e401d2f059"
format_stamp: "Formatted at 2026-09-29 15:58:19 on dist-test-slave-b69v"
server_key: "fe72e9b1d95bee57ebd604e6b2a8840a"
server_key_iv: "e3641bce976856e9cd10b8e388609e35"
server_key_version: "encryptionkey@0"
I20260929 15:58:19.590929 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "575c717d52484db0950744e401d2f059"
format_stamp: "Formatted at 2026-09-29 15:58:19 on dist-test-slave-b69v"
server_key: "fe72e9b1d95bee57ebd604e6b2a8840a"
server_key_iv: "e3641bce976856e9cd10b8e388609e35"
server_key_version: "encryptionkey@0"
I20260929 15:58:19.596220 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 15:58:19.599838 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:19.601935 11817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:19.602190 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "575c717d52484db0950744e401d2f059"
format_stamp: "Formatted at 2026-09-29 15:58:19 on dist-test-slave-b69v"
server_key: "fe72e9b1d95bee57ebd604e6b2a8840a"
server_key_iv: "e3641bce976856e9cd10b8e388609e35"
server_key_version: "encryptionkey@0"
I20260929 15:58:19.602479 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:19.625854 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:19.630581 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:19.630919 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:19.632105 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:19.633661 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:19.633781 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:19.633944 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:19.634020 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:19.673259 11817 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:39409
I20260929 15:58:19.673354 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:39409 every 8 connection(s)
I20260929 15:58:19.675174 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:19.682466 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11817
I20260929 15:58:19.682739 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:19.683703 5588 external_mini_cluster.cc:1428] Setting key d458c39bf371c47dc1fc2ecc9882ae20
Sep 29 15:58:19 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697499, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:19.702909 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.678863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:48323 (local address 127.5.117.62:39627)
0929 15:58:19.679479 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.679485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.681019 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 15:58:19.684458 (+ 3439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.684465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.684576 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.684742 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.687237 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.688109 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.690182 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.690653 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.690824 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.695520 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:19.695527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:19.695531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:19.695579 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:19.699039 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.700205 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.700217 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.700219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.700365 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.700898 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.700909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.700911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.701228 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:19.701394 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.702239 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.702514 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":322,"threads_started":1}
I20260929 15:58:19.704665 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.679632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:48323)
0929 15:58:19.680780 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:19.680835 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:19.683857 (+ 3022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:19.685142 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:19.685158 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:19.686196 (+ 1038us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:19.687007 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.687030 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.688362 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.688367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.689947 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.689968 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.690966 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.690974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.691275 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.692253 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:19.692288 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:19.695131 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:19.699349 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:19.699359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:19.699376 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:19.699990 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:19.700528 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:19.700533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:19.700538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:19.700762 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:19.701368 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:19.701377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:19.701803 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 15:58:19.702112 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 15:58:19.702717 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":440,"threads_started":1}
I20260929 15:58:19.707007 11945 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:19.707592 11945 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:19.708902 11945 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 15:58:19.712373 11756 ts_manager.cc:194] Registered new tserver with Master: 575c717d52484db0950744e401d2f059 (127.5.117.1:39409)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.2@KRBTEST.COM" created.
I20260929 15:58:19.714438 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:48323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
I20260929 15:58:19.748440 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:20.125783 11951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:20.126151 11951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:20.126329 11951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:20.133827 11951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:20.134156 11951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:20.134546 11951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:20.146301 11951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:20.147894 11951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:20.149641 11951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:20.166411 11956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:20.166378 11957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:20.166602 11959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:20.167529 11951 server_base.cc:1034] running on GCE node
I20260929 15:58:20.168292 11951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:20.169278 11951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:20.170521 11951 hybrid_clock.cc:648] HybridClock initialized: now 1790697500170465 us; error 48 us; skew 500 ppm
I20260929 15:58:20.170881 11951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:20 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697500, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:20.174489 11951 init.cc:377] Logged in from keytab as oryx/127.5.117.2@KRBTEST.COM (short username oryx)
I20260929 15:58:20.176186 11951 webserver.cc:460] Webserver started at http://127.5.117.2:40217/ using document root <none> and password file <none>
I20260929 15:58:20.176894 11951 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:20.176996 11951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:20.177304 11951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:20.180362 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "773db94dadc942cbb7e44016c155608c"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "4f979fb07415fe6b9b06db93d33aa805"
server_key_iv: "ae1cfd3d3655ed248f431bcfd867bcdc"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.181109 11951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "773db94dadc942cbb7e44016c155608c"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "4f979fb07415fe6b9b06db93d33aa805"
server_key_iv: "ae1cfd3d3655ed248f431bcfd867bcdc"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.186348 11951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:20.189810 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.191097 11951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:58:20.191280 11951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "773db94dadc942cbb7e44016c155608c"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "4f979fb07415fe6b9b06db93d33aa805"
server_key_iv: "ae1cfd3d3655ed248f431bcfd867bcdc"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.191469 11951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:20.210409 11951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:20.214665 11951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:20.214954 11951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:20.215899 11951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:20.217195 11951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:20.217275 11951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.217442 11951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:20.217490 11951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.251829 11951 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:36947
I20260929 15:58:20.251868 12078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:36947 every 8 connection(s)
I20260929 15:58:20.253401 11951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:20.263134 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 11951
I20260929 15:58:20.263345 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:20.263927 5588 external_mini_cluster.cc:1428] Setting key 65bdb59a5e3fd441b12cf1b9f910822f
Sep 29 15:58:20 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697500, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:20.275113 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.256251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:52015 (local address 127.5.117.62:39627)
0929 15:58:20.256723 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:20.256728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:20.257644 (+ 916us) server_negotiation.cc:407] Connection header received
0929 15:58:20.259558 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:20.259563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:20.259659 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:20.259798 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:20.261343 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.262076 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.263739 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.264331 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.264468 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:20.268802 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:20.268807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:20.268811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:20.268843 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:20.271431 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.272412 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.272416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.272418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.272547 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.273175 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.273183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.273185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.273490 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:20.273661 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:20.274577 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 15:58:20.274799 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260929 15:58:20.276789 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.256794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:52015)
0929 15:58:20.257488 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:20.257531 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:20.259284 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:20.260038 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:20.260087 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:20.260602 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:20.261179 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:20.261196 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.262227 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.262231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:20.263592 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:20.263605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.264553 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.264562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:20.264860 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:20.265605 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:20.265632 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:20.268579 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:20.271569 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:20.271579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:20.271598 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:20.272295 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:20.272660 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:20.272667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:20.272674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:20.272936 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:20.273648 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:20.273658 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:20.274121 (+ 463us) client_negotiation.cc:769] Sending connection context
0929 15:58:20.274450 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 15:58:20.275102 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":276,"threads_started":1}
I20260929 15:58:20.278975 12079 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:20.279381 12079 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:20.280335 12079 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
I20260929 15:58:20.282527 11756 ts_manager.cc:194] Registered new tserver with Master: 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:20.283468 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:52015
WARNING: no policy specified for oryx/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
I20260929 15:58:20.319751 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:20.692274 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:20.692677 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:20.692863 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:20.700963 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:20.701170 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:20.701370 12085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:20.712500 12085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:20.714294 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:20.716018 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:58:20.718868 11945 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
W20260929 15:58:20.732931 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:20.733086 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:20.733531 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:20.734397 12085 server_base.cc:1034] running on GCE node
I20260929 15:58:20.735124 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:20.736454 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:20.737699 12085 hybrid_clock.cc:648] HybridClock initialized: now 1790697500737609 us; error 84 us; skew 500 ppm
I20260929 15:58:20.738061 12085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:20 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697500, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:20.741885 12085 init.cc:377] Logged in from keytab as oryx/127.5.117.3@KRBTEST.COM (short username oryx)
I20260929 15:58:20.743603 12085 webserver.cc:460] Webserver started at http://127.5.117.3:43799/ using document root <none> and password file <none>
I20260929 15:58:20.744462 12085 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:20.744567 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:20.744856 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:20.748039 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "1d33408078f84f10818ad8ff3dd6e755"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "d6d8920696b15346804a491b5a333c99"
server_key_iv: "330c4cb60340dd57f9c169a3adecd287"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.749132 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "1d33408078f84f10818ad8ff3dd6e755"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "d6d8920696b15346804a491b5a333c99"
server_key_iv: "330c4cb60340dd57f9c169a3adecd287"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.755102 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:58:20.759138 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.760994 12085 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:20.761199 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "1d33408078f84f10818ad8ff3dd6e755"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "d6d8920696b15346804a491b5a333c99"
server_key_iv: "330c4cb60340dd57f9c169a3adecd287"
server_key_version: "encryptionkey@0"
I20260929 15:58:20.761417 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:20.781430 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:20.787073 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:20.787441 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:20.788714 12085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:20.790257 12085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:20.790366 12085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.790526 12085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:20.790602 12085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:20.828853 12085 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34643
I20260929 15:58:20.828948 12212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34643 every 8 connection(s)
I20260929 15:58:20.830340 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:20.834450 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12085
I20260929 15:58:20.834679 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:20.835215 5588 external_mini_cluster.cc:1428] Setting key fcf2b82cbc9b796caa606331701916b3
Sep 29 15:58:20 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697500, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:20.852689 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.833582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:49467 (local address 127.5.117.62:39627)
0929 15:58:20.834151 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:20.834160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:20.835357 (+ 1197us) server_negotiation.cc:407] Connection header received
0929 15:58:20.837599 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:20.837606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:20.837689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:20.837796 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:20.839799 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.840707 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.842311 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.842743 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.842840 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:20.846505 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:20.846509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:20.846513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:20.846547 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:20.849535 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.850395 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.850401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.850404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.850578 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.851074 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.851079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.851082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.851412 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:20.851540 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:20.852218 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 15:58:20.852392 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260929 15:58:20.853863 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.834185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:49467)
0929 15:58:20.835083 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:20.835142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:20.837299 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:20.838085 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:20.838096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:20.838926 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:20.839566 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:20.839584 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.840911 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.840920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:20.842185 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:20.842197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.842990 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.842997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:20.843209 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:20.843962 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:20.843989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:20.846313 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:20.849648 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:20.849656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:20.849669 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:20.850277 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:20.850692 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:20.850700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:20.850704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:20.850957 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:20.851526 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:20.851533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:20.851840 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 15:58:20.852141 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 15:58:20.852530 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":386,"threads_started":1}
I20260929 15:58:20.855248 12213 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:20.855526 12213 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:20.856359 12213 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
I20260929 15:58:20.857888 11756 ts_manager.cc:194] Registered new tserver with Master: 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643)
I20260929 15:58:20.858784 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:49467
I20260929 15:58:20.863251 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.117.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.61.keytab.
Entry for principal oryx/127.5.117.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.61.keytab.
I20260929 15:58:20.919982 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:38763
--webserver_interface=127.5.117.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.61.keytab
--principal=oryx/127.5.117.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39627,127.5.117.61:38763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:21.286346 12079 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
W20260929 15:58:21.307507 12219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:21.307901 12219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:21.308131 12219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:21.315373 12219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:21.315510 12219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:21.315606 12219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:21.315702 12219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:21.315788 12219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:21.325551 12219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39627,127.5.117.61:38763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.61.keytab
--principal=oryx/127.5.117.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:38763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.117.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
W20260929 15:58:21.325942 12219 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.117.62:39627,127.5.117.61:38763'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 15:58:21.327152 12219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:21.328747 12219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:21.341058 12224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:21.341030 12227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:21.340986 12225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:21.342123 12219 server_base.cc:1034] running on GCE node
I20260929 15:58:21.342833 12219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:21.344105 12219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:21.345342 12219 hybrid_clock.cc:648] HybridClock initialized: now 1790697501345287 us; error 38 us; skew 500 ppm
I20260929 15:58:21.345672 12219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697501, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:21.348994 12219 init.cc:377] Logged in from keytab as oryx/127.5.117.61@KRBTEST.COM (short username oryx)
I20260929 15:58:21.350597 12219 webserver.cc:460] Webserver started at http://127.5.117.61:41413/ using document root <none> and password file <none>
I20260929 15:58:21.351226 12219 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:21.351307 12219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:21.351562 12219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:21.354879 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/instance:
uuid: "7f69514419264b078027313a5e037933"
format_stamp: "Formatted at 2026-09-29 15:58:21 on dist-test-slave-b69v"
server_key: "351510c74137443f934fae9883d103a9"
server_key_iv: "992fe9e70a5f84fac7410843fc08cc92"
server_key_version: "encryptionkey@0"
I20260929 15:58:21.355674 12219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal/instance:
uuid: "7f69514419264b078027313a5e037933"
format_stamp: "Formatted at 2026-09-29 15:58:21 on dist-test-slave-b69v"
server_key: "351510c74137443f934fae9883d103a9"
server_key_iv: "992fe9e70a5f84fac7410843fc08cc92"
server_key_version: "encryptionkey@0"
I20260929 15:58:21.360694 12219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 15:58:21.364101 12234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:21.365478 12219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:58:21.365646 12219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "7f69514419264b078027313a5e037933"
format_stamp: "Formatted at 2026-09-29 15:58:21 on dist-test-slave-b69v"
server_key: "351510c74137443f934fae9883d103a9"
server_key_iv: "992fe9e70a5f84fac7410843fc08cc92"
server_key_version: "encryptionkey@0"
I20260929 15:58:21.365854 12219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:21.385421 12219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:21.389398 12219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:21.389640 12219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:21.407946 12219 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.61:38763
I20260929 15:58:21.408016 12285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.61:38763 every 8 connection(s)
I20260929 15:58:21.409257 12219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
I20260929 15:58:21.412788 12286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:21.416987 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12219
I20260929 15:58:21.420816 12286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697501, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.61@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:21.441555 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.419438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.5.117.61:38763)
0929 15:58:21.420207 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.420221 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.420272 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:58:21.422034 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.422057 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.422433 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.422811 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.424667 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.426827 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.428476 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.429448 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.429601 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.433991 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.434001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.434011 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.434039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.436703 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.437549 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.437556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.437563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.437753 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.438374 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.438379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.438382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.439056 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.439361 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.439858 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.440265 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":333,"threads_started":1}
I20260929 15:58:21.442180 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.424477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.5.117.62:39627)
0929 15:58:21.425114 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.425120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.425633 (+ 513us) server_negotiation.cc:407] Connection header received
0929 15:58:21.425826 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.425829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.425911 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.426006 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.427364 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.428568 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.430225 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.430643 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.430783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.434521 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.434530 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.434536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.434588 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.437629 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.438521 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.438527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.438530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.438678 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.439163 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.439168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.439171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.439633 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.439789 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.441559 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.441821 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":316,"threads_started":1}
I20260929 15:58:21.442441 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.424480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:57692)
0929 15:58:21.425459 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.425505 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.425692 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.426092 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.426098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.426902 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.427188 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.427196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.428696 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.428701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.430087 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.430096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.430865 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.430871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.431027 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.431596 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.431620 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.434303 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.437805 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.437815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.437820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.438383 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.438825 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.438830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.438834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.439001 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.439860 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.439866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.440022 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.440256 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.440521 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":586,"threads_started":1}
I20260929 15:58:21.444607 12286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697501, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.61@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
I20260929 15:58:21.461582 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.445248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:48938)
0929 15:58:21.445388 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.445406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.445493 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.445807 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.445811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.446136 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.446332 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.446342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.447534 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.447539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.448602 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.448613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.449372 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.449376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.449566 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.455226 (+ 5660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.455242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.457162 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.459847 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.459850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.459852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.460215 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.460500 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.460503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.460505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.460610 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.461010 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.461012 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.461115 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.461203 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.461371 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":4}
I20260929 15:58:21.461663 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.445436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.5.117.61:38763)
0929 15:58:21.445584 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.445588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.445601 (+ 13us) server_negotiation.cc:407] Connection header received
0929 15:58:21.445636 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.445647 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.445690 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.445794 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.446480 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.447378 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.448761 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.449249 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.449367 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.457316 (+ 7949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.457320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.457323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.457348 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.459747 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.460301 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.460304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.460306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.460400 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.460699 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.460702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.460703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.460925 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.461020 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.461268 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.461462 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":47}
I20260929 15:58:21.466722 12286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Bootstrap starting.
I20260929 15:58:21.470261 12286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:21.471350 12286 log.cc:824] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:21.474100 12286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: No bootstrap required, opened a new log
I20260929 15:58:21.481564 12286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } }
I20260929 15:58:21.481882 12286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:21.481956 12286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f69514419264b078027313a5e037933, State: Initialized, Role: FOLLOWER
I20260929 15:58:21.482465 12286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } }
I20260929 15:58:21.483258 12293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } } }
I20260929 15:58:21.483592 12293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:21.484375 12286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:58:21.493453 12304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:58:21.493600 12304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:58:21.496755 12219 master_runner.cc:428] Detected that this master 7f69514419264b078027313a5e037933 is joining an existing cluster
I20260929 15:58:21.496865 12219 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.117.61:38763
I20260929 15:58:21.522770 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.501510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.5.117.62:39627)
0929 15:58:21.501689 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.501695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.502463 (+ 768us) server_negotiation.cc:407] Connection header received
0929 15:58:21.502834 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.502849 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.503011 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.503198 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.508585 (+ 5387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.509818 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.512317 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.512949 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.513126 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.514947 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.514956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.514965 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.515000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.519278 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520262 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.520269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.520273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.520428 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520918 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.520924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.520928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.521332 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.521493 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.522128 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.522373 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 15:58:21.522837 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.502754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.5.117.61:38763)
0929 15:58:21.502950 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.502956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.505418 (+ 2462us) server_negotiation.cc:407] Connection header received
0929 15:58:21.505560 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.505565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.505636 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.505742 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.508998 (+ 3256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.510023 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.511649 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.512452 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.512570 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.515131 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.515139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.515145 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.515175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.519361 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520347 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.520356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.520361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.520526 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520967 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.520975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.520985 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.521403 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.521554 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.522126 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.522386 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":53}
I20260929 15:58:21.524169 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.502473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:48944)
0929 15:58:21.505295 (+ 2822us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.505317 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.505443 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.505908 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.505915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.506740 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.508850 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.508862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.510276 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.510281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.511507 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.511521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.512645 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.512652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.512844 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.513627 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.513656 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.514948 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.519506 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.519514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.519520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.520211 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.520671 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.520684 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.520859 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.521628 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.521634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.521819 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.523090 (+ 1271us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.523434 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2659,"mutex_wait_us":61,"thread_start_us":2476,"threads_started":1}
I20260929 15:58:21.524981 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.501520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:57706)
0929 15:58:21.502304 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.502330 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.502588 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.503470 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.503477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.504719 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.508436 (+ 3717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.508446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.509963 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.509969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.512143 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.512158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.513162 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.513169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.513325 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.513927 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.513943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.514762 (+ 819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.519440 (+ 4678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.519446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.519451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.520116 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.520603 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.520608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.520611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.520785 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.521592 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.521598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.521835 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.523488 (+ 1653us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.523820 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":86,"thread_start_us":440,"threads_started":1}
WARNING: no policy specified for oryx/127.5.117.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.60@KRBTEST.COM" created.
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.62@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
I20260929 15:58:21.559481 12289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.534368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.5.117.61:38763)
0929 15:58:21.540395 (+ 6027us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.540405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.540479 (+ 74us) server_negotiation.cc:407] Connection header received
0929 15:58:21.540922 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.540930 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.541068 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.541184 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.542825 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.543992 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.546162 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.547058 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.547211 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.551889 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.551895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.551900 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.551939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.555939 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.556927 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.556935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.556940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.557109 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.557591 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.557596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.557600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.558001 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.558168 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.558804 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.559063 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5889}
I20260929 15:58:21.559571 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.534243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:48946)
0929 15:58:21.540319 (+ 6076us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.540383 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.540679 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.541253 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.541262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.542258 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.542615 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.542628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.544213 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.544218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.545948 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.545964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.547257 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.547264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.547472 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.548322 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.548349 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.551687 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.556106 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.556121 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.556132 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.556787 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.557243 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.557247 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.557252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.557449 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.558165 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.558175 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.558424 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.558661 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.559073 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5731,"thread_start_us":5424,"threads_started":1}
I20260929 15:58:21.562292 11756 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.117.61:38763: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7f69514419264b078027313a5e037933"
member_type: NON_VOTER
last_known_addr {
host: "127.5.117.61"
port: 38763
}
attrs {
promote: true
}
}
dest_uuid: "ff9b4e17af224cc69aca06fe48edd8b1"
cas_config_opid_index: -1
I20260929 15:58:21.563767 11756 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.60.keytab.
Entry for principal oryx/127.5.117.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.60.keytab.
I20260929 15:58:21.568423 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:40857
--webserver_interface=127.5.117.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.60.keytab
--principal=oryx/127.5.117.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:21.576428 12316 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7f69514419264b078027313a5e037933 (127.5.117.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } }
I20260929 15:58:21.577436 12261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:21.581676 12261 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 15:58:21.581828 12316 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.117.61:38763
I20260929 15:58:21.583263 12219 master.cc:537] Master@127.5.117.61:38763 shutting down...
I20260929 15:58:21.586443 12318 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:58:21.590001 11742 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 15:58:21.590073 12319 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } }
I20260929 15:58:21.590438 12319 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
W20260929 15:58:21.595300 11742 consensus_peers.cc:487] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 -> Peer 7f69514419264b078027313a5e037933 (127.5.117.61:38763): Couldn't send request to peer 7f69514419264b078027313a5e037933. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:21.596989 12219 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:21.597527 12219 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:21.597759 12219 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: stopping tablet replica
I20260929 15:58:21.604398 12219 master.cc:552] Master@127.5.117.61:38763 shutdown complete.
I20260929 15:58:21.604503 12219 master_runner.cc:305] Clearing existing system tablet
I20260929 15:58:21.604946 12219 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:21.605064 12219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:21.607606 12219 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 15:58:21.609351 12323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:21.609719 12219 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:21.609988 12219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 15:58:21.610090 12219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "7f69514419264b078027313a5e037933"
format_stamp: "Formatted at 2026-09-29 15:58:21 on dist-test-slave-b69v"
server_key: "351510c74137443f934fae9883d103a9"
server_key_iv: "992fe9e70a5f84fac7410843fc08cc92"
server_key_version: "encryptionkey@0"
I20260929 15:58:21.613705 12219 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:21.619023 12219 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:58:21.619234 12219 log.cc:1196] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 15:58:21.619691 12219 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Deleting consensus metadata
I20260929 15:58:21.619958 12219 master_runner.cc:315] Copying system tablet from 127.5.117.62:39627
I20260929 15:58:21.622696 12219 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: tablet copy: Beginning tablet copy session from remote peer at address 127.5.117.62:39627
I20260929 15:58:21.636667 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.623740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.5.117.62:39627)
0929 15:58:21.624071 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:21.624082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:21.624610 (+ 528us) server_negotiation.cc:407] Connection header received
0929 15:58:21.624864 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:21.624870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:21.624968 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:21.625083 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:21.625974 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.626784 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.628158 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.628641 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.628785 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.630777 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:21.630782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:21.630785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:21.630820 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:21.633840 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.634502 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.634508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.634512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.634654 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:21.635049 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:21.635055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:21.635058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:21.635367 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:21.635545 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:21.635955 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 15:58:21.636242 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 15:58:21.636698 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:21.623559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:57708)
0929 15:58:21.624487 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:21.624509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:21.624702 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:21.625114 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:21.625118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:21.625582 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:21.625823 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.625833 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.627026 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.627030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.627966 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:21.627977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:21.628799 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:21.628805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:21.628995 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:21.629751 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:21.629770 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:21.630545 (+ 775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:21.633980 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.633990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.633994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.634375 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.634769 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:21.634775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:21.634778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:21.634932 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:21.635497 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:21.635502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:21.635691 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 15:58:21.635850 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 15:58:21.636212 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":679,"threads_started":1}
I20260929 15:58:21.637833 11776 tablet_copy_service.cc:140] P ff9b4e17af224cc69aca06fe48edd8b1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7f69514419264b078027313a5e037933 ({username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:57708)
I20260929 15:58:21.638124 11776 tablet_copy_service.cc:161] P ff9b4e17af224cc69aca06fe48edd8b1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7f69514419264b078027313a5e037933 at {username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:57708: session id = 7f69514419264b078027313a5e037933-00000000000000000000000000000000
I20260929 15:58:21.641682 11776 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:58:21.643792 12219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:58:21.649117 12219 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: tablet copy: Starting download of 0 data blocks...
I20260929 15:58:21.649483 12219 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: tablet copy: Starting download of 1 WAL segments...
I20260929 15:58:21.652168 12219 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:58:21.655874 11776 tablet_copy_service.cc:342] P ff9b4e17af224cc69aca06fe48edd8b1: Request end of tablet copy session 7f69514419264b078027313a5e037933-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:57708
I20260929 15:58:21.656142 11776 tablet_copy_service.cc:433] P ff9b4e17af224cc69aca06fe48edd8b1: ending tablet copy session 7f69514419264b078027313a5e037933-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7f69514419264b078027313a5e037933
I20260929 15:58:21.667623 12219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:21.670688 12330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:21.680780 12331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:21.686121 12333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:21.686326 12219 server_base.cc:1034] running on GCE node
I20260929 15:58:21.686908 12219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:21.687608 12219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:21.688843 12219 hybrid_clock.cc:648] HybridClock initialized: now 1790697501688748 us; error 64 us; skew 500 ppm
I20260929 15:58:21.689144 12219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:21 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697501, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:21.691325 12219 init.cc:377] Logged in from keytab as oryx/127.5.117.61@KRBTEST.COM (short username oryx)
I20260929 15:58:21.692806 12219 webserver.cc:460] Webserver started at http://127.5.117.61:43373/ using document root <none> and password file <none>
I20260929 15:58:21.693151 12219 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:21.693264 12219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:21.695936 12219 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:58:21.698300 12340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:21.698747 12219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 15:58:21.698912 12219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "7f69514419264b078027313a5e037933"
format_stamp: "Formatted at 2026-09-29 15:58:21 on dist-test-slave-b69v"
server_key: "351510c74137443f934fae9883d103a9"
server_key_iv: "992fe9e70a5f84fac7410843fc08cc92"
server_key_version: "encryptionkey@0"
I20260929 15:58:21.699105 12219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:21.717506 12219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:21.721681 12219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:21.759675 12391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.61:38763 every 8 connection(s)
I20260929 15:58:21.759963 12219 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.61:38763
I20260929 15:58:21.761106 12219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
I20260929 15:58:21.768537 12392 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:21.774163 12392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Bootstrap starting.
I20260929 15:58:21.790513 12392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:21.791020 12392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Bootstrap complete.
I20260929 15:58:21.791965 12392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } }
I20260929 15:58:21.792294 12392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f69514419264b078027313a5e037933, State: Initialized, Role: LEARNER
I20260929 15:58:21.792542 12392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } }
I20260929 15:58:21.793404 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } }
I20260929 15:58:21.793761 12392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:21.793792 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:21.810420 12406 catalog_manager.cc:1259] Loaded cluster ID: c2d993f77c1649f4894ceb5b6d42d84c
I20260929 15:58:21.810621 12406 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: loading cluster ID for follower catalog manager: success
I20260929 15:58:21.813833 12406 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: acquiring CA information for follower catalog manager: success
I20260929 15:58:21.816713 12406 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:58:21.861809 12213 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
W20260929 15:58:22.064495 12317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:22.064949 12317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:22.065109 12317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:22.073910 12317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:22.074090 12317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:22.074213 12317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:22.074316 12317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:22.074409 12317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:22.086666 12317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.60.keytab
--principal=oryx/127.5.117.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.117.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:22.088253 12317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:22.089759 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:22.105422 12411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:22.105409 12412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:22.105409 12414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:22.106842 12317 server_base.cc:1034] running on GCE node
I20260929 15:58:22.107748 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:22.109457 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:22.110764 12317 hybrid_clock.cc:648] HybridClock initialized: now 1790697502110692 us; error 38 us; skew 500 ppm
I20260929 15:58:22.111352 12317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697502, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:22.115130 12317 init.cc:377] Logged in from keytab as oryx/127.5.117.60@KRBTEST.COM (short username oryx)
I20260929 15:58:22.117238 12317 webserver.cc:460] Webserver started at http://127.5.117.60:34477/ using document root <none> and password file <none>
I20260929 15:58:22.118048 12317 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:22.118160 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:22.118448 12317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:22.121935 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/instance:
uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
format_stamp: "Formatted at 2026-09-29 15:58:22 on dist-test-slave-b69v"
server_key: "8383f7b2145b0e595a0d73744be961a5"
server_key_iv: "16fe0451e2810cee1d10e7c767257520"
server_key_version: "encryptionkey@0"
I20260929 15:58:22.122713 12317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal/instance:
uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
format_stamp: "Formatted at 2026-09-29 15:58:22 on dist-test-slave-b69v"
server_key: "8383f7b2145b0e595a0d73744be961a5"
server_key_iv: "16fe0451e2810cee1d10e7c767257520"
server_key_version: "encryptionkey@0"
I20260929 15:58:22.129251 12317 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 15:58:22.133255 12421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:22.134738 12317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:22.135022 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
format_stamp: "Formatted at 2026-09-29 15:58:22 on dist-test-slave-b69v"
server_key: "8383f7b2145b0e595a0d73744be961a5"
server_key_iv: "16fe0451e2810cee1d10e7c767257520"
server_key_version: "encryptionkey@0"
I20260929 15:58:22.135252 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:22.157207 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:22.163275 12317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:22.163651 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:22.191119 12317 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.60:40857
I20260929 15:58:22.191250 12472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.60:40857 every 8 connection(s)
I20260929 15:58:22.193495 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
I20260929 15:58:22.198829 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12317
I20260929 15:58:22.200132 12476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:22.204437 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.193408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.5.117.61:38763)
0929 15:58:22.194373 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.194382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.194819 (+ 437us) server_negotiation.cc:407] Connection header received
0929 15:58:22.195256 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.195265 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.195427 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.195587 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:22.196484 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.198013 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.200459 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.202300 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.202503 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.202819 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.203624 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.203857 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":510,"threads_started":1}
I20260929 15:58:22.205613 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.193129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:50636)
0929 15:58:22.194587 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.194616 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.194869 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.195972 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.195977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.196008 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:22.196299 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.196312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.198182 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.198188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.200277 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.200303 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.202997 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.203007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.203310 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.203408 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.204786 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.205037 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":1062,"threads_started":1}
I20260929 15:58:22.207053 12367 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 15:58:22.207929 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9b4e17af224cc69aca06fe48edd8b1. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } }
I20260929 15:58:22.208305 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:22.214180 12476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697502, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.60@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:22.240003 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.219398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.5.117.62:39627)
0929 15:58:22.220235 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.220244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.220947 (+ 703us) server_negotiation.cc:407] Connection header received
0929 15:58:22.221241 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.221252 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.221329 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.221452 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.223493 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.224803 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.226701 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.227586 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.227795 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.232943 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.232952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.232960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.233036 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.236520 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.237506 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.237511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.237514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.237761 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.238255 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.238259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.238261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.238667 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.238812 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.239480 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.239710 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":531,"threads_started":1}
I20260929 15:58:22.240541 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.219314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:47668)
0929 15:58:22.220771 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.220825 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.221072 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.221571 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.221579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.222792 (+ 1213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.223159 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.223172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.224939 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.224945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.226501 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.226513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.227977 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.227984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.228277 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.229078 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.229114 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.232649 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.236696 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.236711 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.236721 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.237363 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.237917 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.237923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.237926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.238131 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.238881 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.238887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.239127 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.239346 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.239723 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":954,"threads_started":1}
I20260929 15:58:22.240563 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.201841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.5.117.60:40857)
0929 15:58:22.214078 (+ 12237us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.214095 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.214149 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:58:22.216874 (+ 2725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.216912 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.217502 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.218052 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.220295 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.221938 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.224124 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.225069 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.225298 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.231202 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.231213 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.231227 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.231274 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.235360 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.236080 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.236089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.236097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.236293 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.236992 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.236995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.236997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.237710 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.238078 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.238576 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.238993 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11950,"thread_start_us":11672,"threads_started":1}
I20260929 15:58:22.242201 12476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697502, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.60@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
I20260929 15:58:22.257967 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.242899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:50652)
0929 15:58:22.243141 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.243158 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.243232 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.243843 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.243846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.244414 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.244603 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.244612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.245709 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.245714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.246773 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.246785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.247532 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.247541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.247732 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.248672 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.248690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.251256 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.255068 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.255074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.255078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.255557 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.256041 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.256048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.256054 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.256245 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.256887 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.256892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.257062 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.257215 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.257566 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 15:58:22.257967 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.243209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.5.117.61:38763)
0929 15:58:22.243410 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.243414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.243429 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:58:22.243477 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.243480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.243579 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.243679 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.244807 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.245537 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.246938 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.247357 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.247458 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.251490 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.251495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.251498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.251540 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.254916 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.255705 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.255710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.255719 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.255862 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.256365 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.256371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.256374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.256692 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.256931 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.257323 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.257576 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:58:22.259557 12476 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697502, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.60@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
I20260929 15:58:22.273975 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.260694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.5.117.60:40857)
0929 15:58:22.260867 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.260880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.261081 (+ 201us) server_negotiation.cc:407] Connection header received
0929 15:58:22.261229 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.261233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.261295 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.261384 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.262217 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.262960 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.264241 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.264779 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.264891 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.268297 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.268301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.268305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.268332 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.271128 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.271754 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.271758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.271760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.271876 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.272424 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.272429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.272432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.272759 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.272922 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.273300 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.273630 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":61}
I20260929 15:58:22.274000 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.260748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.0.0.1:36822)
0929 15:58:22.260971 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.260995 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.261123 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.261503 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.261508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.261914 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.262068 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.262076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.263139 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.263143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.264030 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.264041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.264919 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.264923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.265108 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.265826 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.265844 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.268101 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.271271 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.271275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.271277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.271636 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.272000 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.272004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.272006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.272307 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.272878 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.272882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.273056 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.273202 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.273490 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 15:58:22.279955 12476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Bootstrap starting.
I20260929 15:58:22.283948 12476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:22.285251 12476 log.cc:824] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:22.289027 12476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: No bootstrap required, opened a new log
I20260929 15:58:22.298430 12476 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } }
I20260929 15:58:22.298827 12476 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:22.299080 12476 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ef8ba1e54a4949a69fb8b18158ca44, State: Initialized, Role: FOLLOWER
I20260929 15:58:22.299862 12476 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } }
I20260929 15:58:22.301203 12483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } } }
I20260929 15:58:22.301764 12483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:22.302457 12476 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:58:22.312605 12494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:58:22.312760 12494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:58:22.317863 12317 master_runner.cc:428] Detected that this master f6ef8ba1e54a4949a69fb8b18158ca44 is joining an existing cluster
I20260929 15:58:22.317998 12317 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.117.60:40857
I20260929 15:58:22.342633 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.322761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.5.117.62:39627)
0929 15:58:22.323515 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.323527 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.324842 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 15:58:22.325152 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.325160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.325267 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.325355 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.327020 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.328262 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.330882 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.331676 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.331796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.334907 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.334914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.334919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.334965 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.338821 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.339813 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.339819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.339822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.339986 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.340653 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.340658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.340662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.341094 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.341237 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.341968 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.342215 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553}
I20260929 15:58:22.343815 12500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.322852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:47678)
0929 15:58:22.324711 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.324734 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.324967 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.325592 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.325597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.326431 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.326841 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.326856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.328471 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.328476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.330536 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.330568 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.332261 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.332268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.332512 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.333360 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.333383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.334656 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.338997 (+ 4341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.339004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.339009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.339659 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.340170 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.340178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.340182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.340504 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.341334 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.341339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.341591 (+ 252us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.342913 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.343319 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1706,"thread_start_us":1543,"threads_started":1}
I20260929 15:58:22.347895 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11817
I20260929 15:58:22.353936 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.323737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.5.117.60:40857)
0929 15:58:22.323983 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.323992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.334273 (+ 10281us) server_negotiation.cc:407] Connection header received
0929 15:58:22.334448 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.334454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.334551 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.334660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.336762 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.338105 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.341396 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.342141 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.342235 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.345622 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.345630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.345634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.345666 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.350006 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.351632 (+ 1626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.351639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.351642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.351811 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.352246 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.352251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.352254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.352586 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.352705 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.353305 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.353568 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:58:22.353950 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.325588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.5.117.61:38763)
0929 15:58:22.325842 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.325852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.335094 (+ 9242us) server_negotiation.cc:407] Connection header received
0929 15:58:22.335353 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.335361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.335464 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.335553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.337377 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.338768 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.341405 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.342118 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.342223 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.345649 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.345654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.345659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.345713 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.350096 (+ 4383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.350858 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.350866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.350872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.351544 (+ 672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.352107 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.352112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.352115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.352492 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.352641 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.353312 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.353569 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:58:22.355055 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.325017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:50656)
0929 15:58:22.334983 (+ 9966us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.335000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.335191 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.335824 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.335829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.336786 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.337190 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.337201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.338936 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.338942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.341248 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.341263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.342536 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.342544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.342735 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.343825 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.343857 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.345443 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.350250 (+ 4807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.350254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.350257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.350726 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.351789 (+ 1063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.351793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.351796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.351950 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.352725 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.352729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.353026 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.354215 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.354554 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9798,"mutex_wait_us":190,"thread_start_us":10054,"threads_started":1}
I20260929 15:58:22.355741 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.322899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.0.0.1:36838)
0929 15:58:22.334112 (+ 11213us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.334138 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.334345 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.335245 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.335251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.335932 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.336607 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.336620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.338936 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.338942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.341248 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.341263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.342581 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.342589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.343066 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.344099 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.344149 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.345471 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.350594 (+ 5123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.350598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.350603 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.351508 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.351943 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.351946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.351949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.352113 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.352764 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.352770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.352933 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.354549 (+ 1616us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.354937 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11032,"mutex_wait_us":359,"thread_start_us":11964,"threads_started":1}
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.62@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.1@KRBTEST.COM
I20260929 15:58:22.368813 12474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.350635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.0.0.1:36840)
0929 15:58:22.350833 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.350851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.350946 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.352581 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.352589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.353472 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.353735 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.353751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.356348 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.356354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.357806 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.357822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.358907 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.358914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.359109 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.359721 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.359744 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.362244 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.366276 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.366281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.366285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.366658 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.367043 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.367048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.367052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.367187 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.367821 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.367825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.367976 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.368148 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.368427 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:58:22.369736 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.351150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.5.117.60:40857)
0929 15:58:22.351984 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.351993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.352019 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:22.352209 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.352218 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.352331 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.353991 (+ 1660us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.354489 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.356189 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.357998 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.358715 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.359405 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.362472 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.362477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.362480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.362506 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.366142 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.366777 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.366784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.366788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.366925 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.367309 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.367315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.367318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.367676 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.367837 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.369087 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.369316 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":634,"thread_start_us":465,"threads_started":1}
I20260929 15:58:22.371297 11756 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.117.60:40857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
member_type: NON_VOTER
last_known_addr {
host: "127.5.117.60"
port: 40857
}
attrs {
promote: true
}
}
dest_uuid: "ff9b4e17af224cc69aca06fe48edd8b1"
cas_config_opid_index: 5
I20260929 15:58:22.372699 11756 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } }
I20260929 15:58:22.374670 12473 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.377043 12367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:58:22.378038 12505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:58:22.379750 12448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:22.381465 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.349628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.5.117.1:39409)
0929 15:58:22.357074 (+ 7446us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.357084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.357118 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:58:22.357200 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.357211 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.357540 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.357789 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.359118 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.360834 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.362753 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.363830 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.363980 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.370019 (+ 6039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.370035 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.370070 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.370124 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.374227 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.375019 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.375030 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.375034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.375210 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.378006 (+ 2796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.378015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.378020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.378642 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.379017 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.380462 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.380795 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7247,"thread_start_us":14423,"threads_started":1}
I20260929 15:58:22.384768 12367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.385771 11940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:22.387707 12473 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.117.60:40857
I20260929 15:58:22.388104 12505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.388355 12317 master.cc:537] Master@127.5.117.60:40857 shutting down...
I20260929 15:58:22.388454 12505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:22.390385 12448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:58:22.391520 12473 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:22.394299 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: NON_VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: true } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.394709 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: LEARNER
W20260929 15:58:22.397146 11741 consensus_peers.cc:487] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 -> Peer f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60:40857): Couldn't send request to peer f6ef8ba1e54a4949a69fb8b18158ca44. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:22.397769 12473 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: attempting to promote NON_VOTER 7f69514419264b078027313a5e037933 to VOTER
I20260929 15:58:22.398717 12473 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } }
I20260929 15:58:22.401139 12367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 LEARNER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 15:58:22.401594 11741 consensus_peers.cc:487] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 -> Peer f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60:40857): Couldn't send request to peer f6ef8ba1e54a4949a69fb8b18158ca44. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:22.401816 12505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 15:58:22.405339 12367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f69514419264b078027313a5e037933 (127.5.117.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.410339 12395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.410627 12395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:22.404634 12510 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f69514419264b078027313a5e037933 (127.5.117.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
W20260929 15:58:22.416791 12427 tablet.cc:2369] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:22.421264 12317 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:22.421705 12317 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:22.421934 12317 tablet_replica.cc:330] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: stopping tablet replica
I20260929 15:58:22.429138 12505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.429569 12505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:22.429960 12505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.430256 12505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:22.430672 12317 master.cc:552] Master@127.5.117.60:40857 shutdown complete.
I20260929 15:58:22.430764 12317 master_runner.cc:305] Clearing existing system tablet
I20260929 15:58:22.431228 12317 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:22.431330 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:22.434712 12317 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:58:22.436589 12513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:22.437237 12317 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:22.437527 12317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:58:22.437633 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
format_stamp: "Formatted at 2026-09-29 15:58:22 on dist-test-slave-b69v"
server_key: "8383f7b2145b0e595a0d73744be961a5"
server_key_iv: "16fe0451e2810cee1d10e7c767257520"
server_key_version: "encryptionkey@0"
I20260929 15:58:22.443535 12317 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:22.451361 12317 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:58:22.451530 12317 log.cc:1196] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 15:58:22.452026 12317 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Deleting consensus metadata
I20260929 15:58:22.452351 12317 master_runner.cc:315] Copying system tablet from 127.5.117.62:39627
I20260929 15:58:22.461879 12317 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: tablet copy: Beginning tablet copy session from remote peer at address 127.5.117.62:39627
I20260929 15:58:22.480496 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.463328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.5.117.62:39627)
0929 15:58:22.463605 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:22.463612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:22.464452 (+ 840us) server_negotiation.cc:407] Connection header received
0929 15:58:22.464686 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:22.464692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:22.464804 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:22.464928 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:22.466266 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.467426 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.469484 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.470181 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.470294 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.473318 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:22.473324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:22.473329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:22.473373 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:22.477096 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.477961 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.477967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.477971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.478131 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:22.478636 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:22.478642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:22.478645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:22.479040 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:22.479206 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:22.479770 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 15:58:22.480016 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 15:58:22.481750 12519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:22.462890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:47680)
0929 15:58:22.464279 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:22.464308 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:22.464519 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:22.465155 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:22.465162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:22.465778 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:22.466067 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.466087 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.467645 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.467656 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.469310 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:22.469325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:22.470582 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:22.470588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:22.470775 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:22.471657 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:22.471680 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:22.473083 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:22.477282 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.477287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.477293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.477810 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.478292 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:22.478297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:22.478301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:22.478502 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:22.479292 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:22.479296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:22.479468 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 15:58:22.480535 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0929 15:58:22.481035 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":986,"threads_started":1}
I20260929 15:58:22.482568 11776 tablet_copy_service.cc:140] P ff9b4e17af224cc69aca06fe48edd8b1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f6ef8ba1e54a4949a69fb8b18158ca44 ({username='oryx', principal='oryx/127.5.117.60@KRBTEST.COM'} at 127.0.0.1:47680)
I20260929 15:58:22.482755 11776 tablet_copy_service.cc:161] P ff9b4e17af224cc69aca06fe48edd8b1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f6ef8ba1e54a4949a69fb8b18158ca44 at {username='oryx', principal='oryx/127.5.117.60@KRBTEST.COM'} at 127.0.0.1:47680: session id = f6ef8ba1e54a4949a69fb8b18158ca44-00000000000000000000000000000000
I20260929 15:58:22.487010 11776 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:58:22.489753 12317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:58:22.497560 12317 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: tablet copy: Starting download of 0 data blocks...
I20260929 15:58:22.497969 12317 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: tablet copy: Starting download of 1 WAL segments...
I20260929 15:58:22.501515 12317 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:58:22.506431 11776 tablet_copy_service.cc:342] P ff9b4e17af224cc69aca06fe48edd8b1: Request end of tablet copy session f6ef8ba1e54a4949a69fb8b18158ca44-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.117.60@KRBTEST.COM'} at 127.0.0.1:47680
I20260929 15:58:22.506606 11776 tablet_copy_service.cc:433] P ff9b4e17af224cc69aca06fe48edd8b1: ending tablet copy session f6ef8ba1e54a4949a69fb8b18158ca44-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f6ef8ba1e54a4949a69fb8b18158ca44
I20260929 15:58:22.520493 12317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:58:22.522719 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11817
W20260929 15:58:22.523725 12520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:22.533401 12523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:22.537567 12317 server_base.cc:1034] running on GCE node
W20260929 15:58:22.537894 12521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:22.538411 12317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:22.539494 12317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:22.540706 12317 hybrid_clock.cc:648] HybridClock initialized: now 1790697502540643 us; error 44 us; skew 500 ppm
I20260929 15:58:22.540956 12317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:22 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697502, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:22.543447 12317 init.cc:377] Logged in from keytab as oryx/127.5.117.60@KRBTEST.COM (short username oryx)
I20260929 15:58:22.549350 12317 webserver.cc:460] Webserver started at http://127.5.117.60:45911/ using document root <none> and password file <none>
I20260929 15:58:22.549801 12317 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:22.549934 12317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:22.552904 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:39409
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:39627
--webserver_port=37003
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:22.553690 12317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:58:22.555673 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:22.556121 12317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 15:58:22.556283 12317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
format_stamp: "Formatted at 2026-09-29 15:58:22 on dist-test-slave-b69v"
server_key: "8383f7b2145b0e595a0d73744be961a5"
server_key_iv: "16fe0451e2810cee1d10e7c767257520"
server_key_version: "encryptionkey@0"
I20260929 15:58:22.556511 12317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:22.569329 12317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:22.572901 12317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:22.608354 12317 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.60:40857
I20260929 15:58:22.608382 12582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.60:40857 every 8 connection(s)
I20260929 15:58:22.609423 12317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
I20260929 15:58:22.616945 12583 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:22.624861 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Bootstrap starting.
I20260929 15:58:22.641273 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:22.641824 12583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: Bootstrap complete.
I20260929 15:58:22.642768 12583 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } }
I20260929 15:58:22.643062 12583 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6ef8ba1e54a4949a69fb8b18158ca44, State: Initialized, Role: LEARNER
I20260929 15:58:22.643333 12583 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } }
I20260929 15:58:22.644568 12586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:22.644918 12583 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:22.645128 12586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:22.663192 12597 catalog_manager.cc:1259] Loaded cluster ID: c2d993f77c1649f4894ceb5b6d42d84c
I20260929 15:58:22.663331 12597 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: loading cluster ID for follower catalog manager: success
I20260929 15:58:22.666123 12597 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: acquiring CA information for follower catalog manager: success
I20260929 15:58:22.667935 12597 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 15:58:23.019508 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.009599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.0.0.1:36846)
0929 15:58:23.012762 (+ 3163us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.012781 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.012923 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.013396 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.013400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.013419 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:23.013621 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.013629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.015617 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.015621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.016983 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.016996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.018628 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.018637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.018831 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.018866 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.018995 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.019149 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2991,"thread_start_us":2859,"threads_started":1}
I20260929 15:58:23.020095 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.009869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.5.117.60:40857)
0929 15:58:23.012562 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.012571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.012889 (+ 318us) server_negotiation.cc:407] Connection header received
0929 15:58:23.013094 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.013100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.013216 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.013377 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:23.013777 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.015448 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.017145 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.018404 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.018581 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.018892 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.019062 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.019258 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2529,"thread_start_us":2341,"threads_started":1}
I20260929 15:58:23.021107 12558 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 15:58:23.021921 12586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9b4e17af224cc69aca06fe48edd8b1. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:23.022311 12586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: LEARNER
W20260929 15:58:23.028259 12535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:23.028584 12535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:23.028765 12535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:23.037109 12535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:23.037272 12535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:23.037467 12535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:23.048303 12535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:39409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=37003
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:23.049763 12535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:23.051270 12535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:23.067205 12607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.067248 12605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.067533 12604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:23.068394 12535 server_base.cc:1034] running on GCE node
I20260929 15:58:23.069254 12535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:23.070519 12535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:23.071774 12535 hybrid_clock.cc:648] HybridClock initialized: now 1790697503071718 us; error 48 us; skew 500 ppm
I20260929 15:58:23.072253 12535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:23.076743 12535 init.cc:377] Logged in from keytab as oryx/127.5.117.1@KRBTEST.COM (short username oryx)
I20260929 15:58:23.078743 12535 webserver.cc:460] Webserver started at http://127.5.117.1:37003/ using document root <none> and password file <none>
I20260929 15:58:23.079705 12535 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:23.079892 12535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:23.087328 12535 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:58:23.091507 12614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.093528 12535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:23.093814 12535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "575c717d52484db0950744e401d2f059"
format_stamp: "Formatted at 2026-09-29 15:58:19 on dist-test-slave-b69v"
server_key: "fe72e9b1d95bee57ebd604e6b2a8840a"
server_key_iv: "e3641bce976856e9cd10b8e388609e35"
server_key_version: "encryptionkey@0"
I20260929 15:58:23.095324 12535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:23.112700 12535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:23.118476 12535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:23.118898 12535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:23.120148 12535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:23.121811 12535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:23.121941 12535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.122092 12535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:23.122285 12535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.160943 12535 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:39409
I20260929 15:58:23.161194 12726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:39409 every 8 connection(s)
I20260929 15:58:23.162835 12535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:23.169332 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12535
I20260929 15:58:23.169826 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11951
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.2@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:23.209283 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.167028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:35505 (local address 127.5.117.61:38763)
0929 15:58:23.179370 (+ 12342us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.179380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.182008 (+ 2628us) server_negotiation.cc:407] Connection header received
0929 15:58:23.184755 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.184761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.184865 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.184961 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.187861 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.189098 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.192670 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.193403 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.193548 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.200375 (+ 6827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.200385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.200392 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.200443 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.204972 (+ 4529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.206050 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.206057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.206061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.206208 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.206680 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.206685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.206690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.207092 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.207239 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.208370 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.208815 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12150,"thread_start_us":12345,"threads_started":1}
I20260929 15:58:23.210793 12599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.169075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:49001 (local address 127.5.117.60:40857)
0929 15:58:23.169411 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.169421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.180534 (+ 11113us) server_negotiation.cc:407] Connection header received
0929 15:58:23.184520 (+ 3986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.184526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.184631 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.184738 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.187862 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.189100 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.192277 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.192949 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.193092 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.199542 (+ 6450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.199549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.199555 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.199599 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.204776 (+ 5177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.205880 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.205889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.205893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.206066 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.206672 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.206677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.206680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.207061 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.207208 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.208381 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.208639 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 15:58:23.210824 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.181095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:38807 (local address 127.5.117.62:39627)
0929 15:58:23.184572 (+ 3477us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.184581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.192285 (+ 7704us) server_negotiation.cc:407] Connection header received
0929 15:58:23.192848 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.192854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.192955 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.193060 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.194902 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.196350 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.198178 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.199024 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.199163 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.203181 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.203190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.203197 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.203241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.207147 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.208006 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.208012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.208016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.208196 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.208848 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.208853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.208856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.209222 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.209378 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.210166 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.210408 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3283,"thread_start_us":3126,"threads_started":1}
I20260929 15:58:23.211069 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.171299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.5.117.2:36947)
0929 15:58:23.185151 (+ 13852us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.185164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.185204 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:58:23.185309 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.185321 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.185635 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.188826 (+ 3191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.189184 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.191278 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.193718 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.194796 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.201842 (+ 7046us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.202180 (+ 338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.202193 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.202214 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.202254 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.206670 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.207573 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.207582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.207585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.207777 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.208271 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.208276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.208279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.208710 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.209032 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.210313 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.210590 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13643,"thread_start_us":13454,"threads_started":1}
I20260929 15:58:23.211944 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.167996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.5.117.1:35505)
0929 15:58:23.181871 (+ 13875us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.181902 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.184598 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.185244 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.185257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.186385 (+ 1128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.187640 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.187675 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.189319 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.189324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.192528 (+ 3204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.192537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.193850 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.193856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.194071 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.194866 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.194900 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.200112 (+ 5212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.205167 (+ 5055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.205174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.205180 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.205890 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.206359 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.206363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.206366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.206540 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.207323 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.207332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.207843 (+ 511us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.209581 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.210217 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13684,"mutex_wait_us":402,"spinlock_wait_cycles":24064,"thread_start_us":11587,"threads_started":1}
I20260929 15:58:23.211966 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.180956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:38807)
0929 15:58:23.192183 (+ 11227us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.192204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.192681 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.193401 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.193406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.194260 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.194687 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.194699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.196557 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.196563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.198053 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.198066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.199241 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.199250 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.199493 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.200227 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.200249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.202944 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.207316 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.207321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.207324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.207852 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.208327 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.208332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.208335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.208680 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.209464 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.209469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.209604 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.209785 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.211148 (+ 1363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11075,"mutex_wait_us":5,"thread_start_us":370,"threads_started":1}
I20260929 15:58:23.212801 12074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:23.213193 12732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.167865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.5.117.1:49001)
0929 15:58:23.180285 (+ 12420us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.180371 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.184148 (+ 3777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.185244 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.185257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.186385 (+ 1128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.187640 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.187675 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.189319 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.189324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.192113 (+ 2789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.192127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.193427 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.193435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.193797 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.194985 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.195010 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.199208 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.204950 (+ 5742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.204962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.204984 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.205726 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.206241 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.206246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.206249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.206524 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.207329 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.207335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.207843 (+ 508us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.212405 (+ 4562us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.212718 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11927,"mutex_wait_us":344,"spinlock_wait_cycles":33920,"thread_start_us":23547,"threads_started":1}
I20260929 15:58:23.213924 12729 heartbeater.cc:347] Connected to a master server at 127.5.117.61:38763
I20260929 15:58:23.214080 12728 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:23.214304 12728 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.214581 12727 heartbeater.cc:347] Connected to a master server at 127.5.117.60:40857
I20260929 15:58:23.214707 12727 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.214310 12729 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.215493 12728 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
I20260929 15:58:23.215808 12727 heartbeater.cc:511] Master 127.5.117.60:40857 requested a full tablet report, sending...
I20260929 15:58:23.216251 12729 heartbeater.cc:511] Master 127.5.117.61:38763 requested a full tablet report, sending...
I20260929 15:58:23.217878 11756 ts_manager.cc:194] Re-registered known tserver with Master: 575c717d52484db0950744e401d2f059 (127.5.117.1:39409)
I20260929 15:58:23.218351 12357 ts_manager.cc:194] Registered new tserver with Master: 575c717d52484db0950744e401d2f059 (127.5.117.1:39409)
I20260929 15:58:23.219491 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:38807
I20260929 15:58:23.221208 12548 ts_manager.cc:194] Registered new tserver with Master: 575c717d52484db0950744e401d2f059 (127.5.117.1:39409)
I20260929 15:58:23.309659 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11951
I20260929 15:58:23.330837 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:36947
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:39627
--webserver_port=40217
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:23.532541 12510 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: NON_VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: true } } }
I20260929 15:58:23.532974 12510 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:23.534428 12741 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: attempting to promote NON_VOTER f6ef8ba1e54a4949a69fb8b18158ca44 to VOTER
I20260929 15:58:23.535284 12741 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } }
I20260929 15:58:23.537786 12558 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 LEARNER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:58:23.538429 12367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Refusing update from remote peer ff9b4e17af224cc69aca06fe48edd8b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:58:23.538517 12740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:58:23.539180 12741 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:58:23.542423 12740 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.543105 12367 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.544328 12558 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.547097 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.547160 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.547444 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:23.547441 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:23.547699 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.547888 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:23.555225 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.556725 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:23.556543 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:23.556955 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 15:58:23.713194 12739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:23.713563 12739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:23.713713 12739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:23.721423 12739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:23.721534 12739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:23.721664 12739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:23.731506 12739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:36947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=40217
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:23.733075 12739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:23.734681 12739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:23.750083 12752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.750134 12755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.750077 12753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:23.751564 12739 server_base.cc:1034] running on GCE node
I20260929 15:58:23.752362 12739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:23.753791 12739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:23.755018 12739 hybrid_clock.cc:648] HybridClock initialized: now 1790697503754938 us; error 75 us; skew 500 ppm
I20260929 15:58:23.755403 12739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:23.759505 12739 init.cc:377] Logged in from keytab as oryx/127.5.117.2@KRBTEST.COM (short username oryx)
I20260929 15:58:23.761448 12739 webserver.cc:460] Webserver started at http://127.5.117.2:40217/ using document root <none> and password file <none>
I20260929 15:58:23.762329 12739 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:23.762503 12739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:23.768985 12739 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 15:58:23.772244 12762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.773768 12739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:58:23.773975 12739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "773db94dadc942cbb7e44016c155608c"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "4f979fb07415fe6b9b06db93d33aa805"
server_key_iv: "ae1cfd3d3655ed248f431bcfd867bcdc"
server_key_version: "encryptionkey@0"
I20260929 15:58:23.775131 12739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:23.789320 12739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:23.794029 12739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:23.794401 12739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:23.795588 12739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:23.797261 12739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:23.797384 12739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.797510 12739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:23.797616 12739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.833014 12739 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:36947
I20260929 15:58:23.833040 12874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:36947 every 8 connection(s)
I20260929 15:58:23.834568 12739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:23.835553 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12739
I20260929 15:58:23.835911 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12085
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.3@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
Sep 29 15:58:23 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
I20260929 15:58:23.882584 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.837077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.5.117.3:34643)
0929 15:58:23.849593 (+ 12516us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.849607 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.850852 (+ 1245us) server_negotiation.cc:407] Connection header received
0929 15:58:23.850974 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.850988 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.851354 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.851573 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.854840 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.857108 (+ 2268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.859758 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.860979 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.861126 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.869566 (+ 8440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.869586 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.869610 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.869676 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.874786 (+ 5110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.876391 (+ 1605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.876402 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.876408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.876859 (+ 451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.877356 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.877362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.877365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.879799 (+ 2434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.880152 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.880541 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.882007 (+ 1466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12221,"thread_start_us":13453,"threads_started":1}
I20260929 15:58:23.884843 12208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:23.888868 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.837831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35579 (local address 127.5.117.62:39627)
0929 15:58:23.854140 (+ 16309us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.854151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.855576 (+ 1425us) server_negotiation.cc:407] Connection header received
0929 15:58:23.859340 (+ 3764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.859358 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.859484 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.859609 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.862558 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.864018 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.873763 (+ 9745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.874309 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.874449 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.879887 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.879894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.879899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.879953 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.884200 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.885433 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.885442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.885448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.885657 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.886289 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.886296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.886302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.886783 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.886955 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.887844 (+ 889us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.888218 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16089,"thread_start_us":15785,"threads_started":1}
I20260929 15:58:23.888913 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.843643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35651 (local address 127.5.117.60:40857)
0929 15:58:23.865980 (+ 22337us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.865990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.866010 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:23.866079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.866084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.866172 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.866295 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.867610 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.870585 (+ 2975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.874306 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.875007 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.875129 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.879887 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.879894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.879899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.879947 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.884132 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.885385 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.885391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.885394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.885547 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.886068 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.886075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.886080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.886487 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.886619 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.888251 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.888495 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22194,"thread_start_us":22063,"threads_started":1}
I20260929 15:58:23.889976 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.843015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.5.117.2:35651)
0929 15:58:23.865180 (+ 22165us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.865203 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.865349 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.866310 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.866314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.867053 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.867429 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.867443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.871005 (+ 3562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.871013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.874120 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.874139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.875382 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.875389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.875586 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.876229 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.876250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.879636 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.884313 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.884323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.884346 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.885232 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.885679 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.885685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.885691 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.885903 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.886706 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.886715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.887280 (+ 565us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.887622 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.889442 (+ 1820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22015,"mutex_wait_us":1265,"thread_start_us":23572,"threads_started":1}
I20260929 15:58:23.890415 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.838536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:35579)
0929 15:58:23.855080 (+ 16544us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.855150 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.858961 (+ 3811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.859966 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.859985 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.861182 (+ 1197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.862348 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.862389 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.864679 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.864687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.873588 (+ 8901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.873605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.874578 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.874588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.874939 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.875782 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.875825 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.879554 (+ 3729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.884369 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.884373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.884377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.885256 (+ 879us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.885846 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.885854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.885860 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.886134 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.887054 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.887061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.887310 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.889120 (+ 1810us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.889686 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16064,"mutex_wait_us":336,"spinlock_wait_cycles":77696,"thread_start_us":15776,"threads_started":1}
I20260929 15:58:23.890807 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.857211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:51993 (local address 127.5.117.61:38763)
0929 15:58:23.865502 (+ 8291us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:23.865512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:23.867168 (+ 1656us) server_negotiation.cc:407] Connection header received
0929 15:58:23.867246 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:23.867251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:23.867359 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:23.868305 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:23.869692 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.870944 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.874768 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.875541 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.875668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.881605 (+ 5937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:23.881613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:23.881621 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:23.881673 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:23.885462 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.886315 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.886320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.886323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.886478 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:23.886923 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:23.886927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:23.886930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:23.887419 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:23.887583 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:23.890156 (+ 2573us) server_negotiation.cc:299] Negotiation successful
0929 15:58:23.890429 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8135,"thread_start_us":8003,"threads_started":1}
I20260929 15:58:23.889945 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:23.857052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.5.117.2:51993)
0929 15:58:23.866917 (+ 9865us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:23.866934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:23.867052 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:23.868501 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:23.868504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:23.869103 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:23.869518 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.869533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.871187 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.871194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.873613 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:23.873622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:23.875944 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:23.875950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:23.876217 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:23.876839 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:23.876863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:23.881370 (+ 4507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:23.885613 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.885618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.885621 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.886179 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.886611 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:23.886615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:23.886619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:23.886798 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:23.887642 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:23.887645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:23.887766 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 15:58:23.887902 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 15:58:23.888297 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9720,"mutex_wait_us":144,"thread_start_us":7771,"threads_started":1}
I20260929 15:58:23.891978 12875 heartbeater.cc:347] Connected to a master server at 127.5.117.60:40857
I20260929 15:58:23.892405 12875 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.892721 12877 heartbeater.cc:347] Connected to a master server at 127.5.117.61:38763
I20260929 15:58:23.892874 12877 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.893463 12876 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:23.893618 12876 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:23.893769 12875 heartbeater.cc:511] Master 127.5.117.60:40857 requested a full tablet report, sending...
I20260929 15:58:23.894013 12877 heartbeater.cc:511] Master 127.5.117.61:38763 requested a full tablet report, sending...
I20260929 15:58:23.894375 12876 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
I20260929 15:58:23.896145 11756 ts_manager.cc:194] Re-registered known tserver with Master: 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:23.896176 12357 ts_manager.cc:194] Registered new tserver with Master: 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:23.896309 12548 ts_manager.cc:194] Registered new tserver with Master: 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:23.897416 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:35579
I20260929 15:58:23.993363 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12085
I20260929 15:58:24.014070 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:34643
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:39627
--webserver_port=43799
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:24.222849 12728 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
W20260929 15:58:24.404942 12888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:24.405303 12888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:24.405781 12888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:24.413960 12888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:24.414101 12888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:24.414275 12888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:24.427629 12888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:34643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=43799
--tserver_master_addrs=127.5.117.62:39627,127.5.117.61:38763,127.5.117.60:40857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:24.429106 12888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:24.430575 12888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:24.447572 12894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:24.447675 12893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:24.448154 12896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:24.449039 12888 server_base.cc:1034] running on GCE node
I20260929 15:58:24.449839 12888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:24.450978 12888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:24.452178 12888 hybrid_clock.cc:648] HybridClock initialized: now 1790697504452129 us; error 33 us; skew 500 ppm
I20260929 15:58:24.452594 12888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:24 dist-test-slave-b69v krb5kdc[11711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:24.457093 12888 init.cc:377] Logged in from keytab as oryx/127.5.117.3@KRBTEST.COM (short username oryx)
I20260929 15:58:24.459326 12888 webserver.cc:460] Webserver started at http://127.5.117.3:43799/ using document root <none> and password file <none>
I20260929 15:58:24.460224 12888 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:24.460399 12888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:24.468012 12888 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:24.471148 12903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.473160 12888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:24.473381 12888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "1d33408078f84f10818ad8ff3dd6e755"
format_stamp: "Formatted at 2026-09-29 15:58:20 on dist-test-slave-b69v"
server_key: "d6d8920696b15346804a491b5a333c99"
server_key_iv: "330c4cb60340dd57f9c169a3adecd287"
server_key_version: "encryptionkey@0"
I20260929 15:58:24.474478 12888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:24.496035 12888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:24.500869 12888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:24.501187 12888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:24.502219 12888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:24.504966 12888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:24.505138 12888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.505275 12888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:24.505367 12888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.555119 12888 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34643
I20260929 15:58:24.555219 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34643 every 8 connection(s)
I20260929 15:58:24.556967 12888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:24.558790 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 12888
Sep 29 15:58:24 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.60@KRBTEST.COM
Sep 29 15:58:24 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.61@KRBTEST.COM
I20260929 15:58:24.620710 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.560657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:33325 (local address 127.5.117.61:38763)
0929 15:58:24.586493 (+ 25836us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.586503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.591484 (+ 4981us) server_negotiation.cc:407] Connection header received
0929 15:58:24.591603 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.591613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.591754 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.591898 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.596972 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.597863 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.601278 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.601814 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.601924 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.608719 (+ 6795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.608726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.608730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.608789 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.612729 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.614172 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.614177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.614181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.614368 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.616701 (+ 2333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.616709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.616715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.617245 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.617438 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.619888 (+ 2450us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.620174 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25654,"thread_start_us":31429,"threads_started":1}
I20260929 15:58:24.622965 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.561058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:59755 (local address 127.5.117.60:40857)
0929 15:58:24.585892 (+ 24834us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.585904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.593448 (+ 7544us) server_negotiation.cc:407] Connection header received
0929 15:58:24.593675 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.593683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.593804 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.593944 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.598076 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.599030 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.601432 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.602318 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.602488 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.607625 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.607633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.607640 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.607697 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.616485 (+ 8788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.618597 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.618605 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.618611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.618807 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.619227 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.619233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.619236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.619568 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.619719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.622324 (+ 2605us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.622559 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24631,"thread_start_us":24865,"threads_started":1}
I20260929 15:58:24.624559 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.561371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.5.117.3:59755)
0929 15:58:24.593269 (+ 31898us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.593305 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.593488 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.594205 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.594210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.595722 (+ 1512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:24.596754 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.596783 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.599194 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.599199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.601194 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.601227 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.602591 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.602596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.602828 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.603763 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:24.603790 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:24.607295 (+ 3505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:24.617570 (+ 10275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.617578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.617585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.618403 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.618943 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.618948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.618952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.619088 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.621025 (+ 1937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:24.621030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:24.621187 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.621372 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.621703 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31664,"mutex_wait_us":191,"spinlock_wait_cycles":21632,"thread_start_us":31254,"threads_started":1}
I20260929 15:58:24.627167 13016 heartbeater.cc:347] Connected to a master server at 127.5.117.60:40857
I20260929 15:58:24.627591 13016 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:24.628360 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.561371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.5.117.3:33325)
0929 15:58:24.587211 (+ 25840us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.587265 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.590926 (+ 3661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.594089 (+ 3163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.594110 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.595736 (+ 1626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:24.596756 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.596787 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.598015 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.598023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.601086 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.601105 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.603134 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.603149 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.603481 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.604381 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:24.604415 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:24.607852 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:24.612905 (+ 5053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.612920 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.612948 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.613795 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.614492 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.614501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.614507 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.614798 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.618786 (+ 3988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:24.618798 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:24.619383 (+ 585us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.619715 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.620445 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25475,"mutex_wait_us":14,"spinlock_wait_cycles":3968,"thread_start_us":25195,"threads_started":1}
I20260929 15:58:24.628901 13016 heartbeater.cc:511] Master 127.5.117.60:40857 requested a full tablet report, sending...
Sep 29 15:58:24 dist-test-slave-b69v krb5kdc[11711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:24.632264 12548 ts_manager.cc:194] Registered new tserver with Master: 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643)
I20260929 15:58:24.632632 13018 heartbeater.cc:347] Connected to a master server at 127.5.117.61:38763
I20260929 15:58:24.632802 13018 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:24.633457 13018 heartbeater.cc:511] Master 127.5.117.61:38763 requested a full tablet report, sending...
I20260929 15:58:24.635459 12357 ts_manager.cc:194] Registered new tserver with Master: 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643)
I20260929 15:58:24.641050 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.588013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:59607)
0929 15:58:24.595265 (+ 7252us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.595293 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.595420 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.609075 (+ 13655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.609080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.609768 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:24.610132 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.610147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.616472 (+ 6325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.616481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.618559 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.618578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.621313 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.621322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.621621 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.623526 (+ 1905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:24.623565 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:24.633236 (+ 9671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:24.638182 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.638187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.638191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.638770 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.639225 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.639230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.639233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.639426 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.640208 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:24.640213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:24.640348 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.640459 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.640706 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7103,"thread_start_us":6625,"threads_started":1}
I20260929 15:58:24.641350 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.588203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:59607 (local address 127.5.117.62:39627)
0929 15:58:24.608671 (+ 20468us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.608678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.608701 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:58:24.608790 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.608797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.608896 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.609044 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.615061 (+ 6017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.616334 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.619519 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.620128 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.620388 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.633498 (+ 13110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.633505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.633509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.633552 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.638014 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.638910 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.638916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.638920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.639091 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.639556 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.639563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.639566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.640082 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.640240 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.640582 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.640819 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":20279,"thread_start_us":20960,"threads_started":1}
I20260929 15:58:24.642220 13017 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39627
I20260929 15:58:24.642362 13017 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:24.642832 13017 heartbeater.cc:511] Master 127.5.117.62:39627 requested a full tablet report, sending...
I20260929 15:58:24.643949 11756 ts_manager.cc:194] Re-registered known tserver with Master: 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643)
I20260929 15:58:24.645300 11756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:59607
I20260929 15:58:24.655143 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.609390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.5.117.62:39627)
0929 15:58:24.622838 (+ 13448us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.622844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.625337 (+ 2493us) server_negotiation.cc:407] Connection header received
0929 15:58:24.625943 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.625953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.626087 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.626220 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.639594 (+ 13374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.640817 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.643343 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.643884 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.644020 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.648368 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.648376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.648382 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.648425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.652262 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.652962 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.652966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.652968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.653094 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.653569 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.653574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.653577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.653849 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.654169 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.654526 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.654741 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13307,"thread_start_us":13474,"threads_started":1}
I20260929 15:58:24.655221 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.610408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.5.117.61:38763)
0929 15:58:24.610962 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.610969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.611751 (+ 782us) server_negotiation.cc:407] Connection header received
0929 15:58:24.612189 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.612197 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.612311 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.612412 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.635011 (+ 22599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.636497 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.637987 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.639699 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.639830 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.647421 (+ 7591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.647428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.647433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.647473 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.651039 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.651738 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.651746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.651752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.651921 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.652332 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.652338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.652341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.652754 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.653022 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.654303 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.654579 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":323,"threads_started":1}
I20260929 15:58:24.659273 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.612160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.5.117.60:40857)
0929 15:58:24.623236 (+ 11076us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.623244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.634973 (+ 11729us) server_negotiation.cc:407] Connection header received
0929 15:58:24.635226 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.635233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.635346 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.635414 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.638445 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.639416 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.645760 (+ 6344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.646484 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.646626 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.652446 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.652453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.652457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.652498 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.656210 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.657043 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.657048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.657052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.657257 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.657705 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.657709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.657711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.657980 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.658446 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.658650 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.658903 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10941,"thread_start_us":20944,"threads_started":1}
I20260929 15:58:24.660056 11756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:24.667749 11756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:24.702366 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.690636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.5.117.3:34643)
0929 15:58:24.692278 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.692291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.692339 (+ 48us) server_negotiation.cc:407] Connection header received
0929 15:58:24.692473 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.692483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.692880 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.693178 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:24.693730 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.695834 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.698730 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.701076 (+ 2346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.701266 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.701539 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.701792 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.701978 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":401,"threads_started":1}
I20260929 15:58:24.702862 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.690155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34643 (local address 127.0.0.1:33382)
0929 15:58:24.691835 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.691855 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.691985 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.693141 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.693148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.693168 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:24.693497 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.693505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.695993 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.695999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.698580 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.698597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.701352 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.701358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.701547 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.701588 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.701738 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.701889 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":1354,"threads_started":1}
I20260929 15:58:24.704640 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.689169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39409 (local address 127.0.0.1:53824)
0929 15:58:24.690154 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.690185 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.690380 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.697179 (+ 6799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.697188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.697215 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:24.697737 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.697747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.699388 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.699393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.701718 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.701730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.703696 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.703704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.703906 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.703943 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.704017 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.704212 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":1,"thread_start_us":564,"threads_started":1}
I20260929 15:58:24.704917 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.691866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36947 (local address 127.0.0.1:44290)
0929 15:58:24.692644 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.692658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.692746 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.695843 (+ 3097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.695849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.695878 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:24.696542 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.696554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.698649 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.698655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.701066 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.701084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.704003 (+ 2919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.704010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.704221 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.704256 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.704336 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.704492 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":507,"threads_started":1}
I20260929 15:58:24.705263 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.692224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.5.117.2:36947)
0929 15:58:24.694896 (+ 2672us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.694903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.694929 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:24.695072 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.695080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.695381 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.695604 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:24.696740 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.698471 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.702143 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.703650 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.703810 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.704602 (+ 792us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.704718 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.704923 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2519,"thread_start_us":2426,"threads_started":1}
I20260929 15:58:24.705972 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.689493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.5.117.1:39409)
0929 15:58:24.696664 (+ 7171us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.696674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.696705 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:58:24.696813 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.696819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.696978 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.697201 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:24.697943 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.699207 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.701901 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.703457 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.705121 (+ 1664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.705361 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.705438 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.705668 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6905,"thread_start_us":6566,"threads_started":1}
I20260929 15:58:24.707963 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 37e462d655244c9abcc136a0124ad95a (DEFAULT_TABLE table=test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:24.708669 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 37e462d655244c9abcc136a0124ad95a (DEFAULT_TABLE table=test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:24.709790 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e462d655244c9abcc136a0124ad95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:24.709365 12662 tablet_service.cc:1467] Processing CreateTablet for tablet 37e462d655244c9abcc136a0124ad95a (DEFAULT_TABLE table=test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:24.709903 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e462d655244c9abcc136a0124ad95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:24.710649 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e462d655244c9abcc136a0124ad95a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:24.724653 13053 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Bootstrap starting.
I20260929 15:58:24.728935 13055 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Bootstrap starting.
I20260929 15:58:24.729367 13054 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Bootstrap starting.
I20260929 15:58:24.730906 13053 tablet_bootstrap.cc:654] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:24.732609 13053 log.cc:824] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:24.733814 13054 tablet_bootstrap.cc:654] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:24.735154 13054 log.cc:824] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:24.736383 13055 tablet_bootstrap.cc:654] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:24.736685 13053 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: No bootstrap required, opened a new log
I20260929 15:58:24.736999 13053 ts_tablet_manager.cc:1397] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 15:58:24.737731 13055 log.cc:824] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:24.739089 13054 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: No bootstrap required, opened a new log
I20260929 15:58:24.739365 13054 ts_tablet_manager.cc:1397] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 15:58:24.740496 13055 tablet_bootstrap.cc:492] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: No bootstrap required, opened a new log
I20260929 15:58:24.740785 13055 ts_tablet_manager.cc:1397] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260929 15:58:24.749114 13053 raft_consensus.cc:348] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.749715 13053 raft_consensus.cc:374] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:24.749507 13055 raft_consensus.cc:348] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.749821 13053 raft_consensus.cc:729] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773db94dadc942cbb7e44016c155608c, State: Initialized, Role: FOLLOWER
I20260929 15:58:24.749913 13055 raft_consensus.cc:374] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:24.750059 13055 raft_consensus.cc:729] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575c717d52484db0950744e401d2f059, State: Initialized, Role: FOLLOWER
I20260929 15:58:24.750685 13055 consensus_queue.cc:260] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.750695 13053 consensus_queue.cc:260] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.751223 13054 raft_consensus.cc:348] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.751782 13054 raft_consensus.cc:374] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:24.751901 13054 raft_consensus.cc:729] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d33408078f84f10818ad8ff3dd6e755, State: Initialized, Role: FOLLOWER
I20260929 15:58:24.752794 13055 ts_tablet_manager.cc:1428] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260929 15:58:24.752857 13054 consensus_queue.cc:260] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.759092 13054 ts_tablet_manager.cc:1428] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260929 15:58:24.763919 13017 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
I20260929 15:58:24.766718 12876 heartbeater.cc:503] Master 127.5.117.62:39627 was elected leader, sending a full tablet report...
I20260929 15:58:24.769047 13053 ts_tablet_manager.cc:1428] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Time spent starting tablet: real 0.032s user 0.016s sys 0.012s
W20260929 15:58:24.829840 13019 tablet.cc:2369] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:24.866441 13059 raft_consensus.cc:482] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:24.866873 13059 raft_consensus.cc:504] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.869488 13059 leader_election.cc:288] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643), 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
W20260929 15:58:24.873481 12886 tablet.cc:2369] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:24.878793 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.870306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:56017 (local address 127.5.117.3:34643)
0929 15:58:24.870597 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.870604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.870882 (+ 278us) server_negotiation.cc:407] Connection header received
0929 15:58:24.871165 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.871174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.871279 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.871389 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:24.872360 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.873338 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.876185 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.877459 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.877599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.877721 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.878195 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.878348 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:58:24.878799 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.869963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34643 (local address 127.5.117.1:56017)
0929 15:58:24.870760 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.870777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.870992 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.871504 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.871510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.871549 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:24.872190 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.872207 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.873474 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.873482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.876005 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.876020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.877700 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.877704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.878001 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.878058 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.878163 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.878303 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":78,"thread_start_us":437,"threads_started":1}
I20260929 15:58:24.879134 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.871205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:33983 (local address 127.5.117.2:36947)
0929 15:58:24.871449 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.871458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.871728 (+ 270us) server_negotiation.cc:407] Connection header received
0929 15:58:24.871865 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.871871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.871963 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.872188 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:24.872826 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.873896 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.876375 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.877767 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.877872 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.877942 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.878455 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.878734 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 15:58:24.879133 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.870850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36947 (local address 127.5.117.1:33983)
0929 15:58:24.871599 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.871613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.871743 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.872224 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.872227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.872246 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:24.872646 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.872658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.874064 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.874069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.876241 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.876250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.878081 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.878086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.878295 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.878324 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.878409 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.878559 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":434,"threads_started":1}
I20260929 15:58:24.880199 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e462d655244c9abcc136a0124ad95a" candidate_uuid: "575c717d52484db0950744e401d2f059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d33408078f84f10818ad8ff3dd6e755" is_pre_election: true
I20260929 15:58:24.880391 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e462d655244c9abcc136a0124ad95a" candidate_uuid: "575c717d52484db0950744e401d2f059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773db94dadc942cbb7e44016c155608c" is_pre_election: true
I20260929 15:58:24.881124 12971 raft_consensus.cc:2454] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575c717d52484db0950744e401d2f059 in term 0.
I20260929 15:58:24.881311 12830 raft_consensus.cc:2454] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575c717d52484db0950744e401d2f059 in term 0.
I20260929 15:58:24.881891 12617 leader_election.cc:302] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d33408078f84f10818ad8ff3dd6e755, 575c717d52484db0950744e401d2f059; no voters:
I20260929 15:58:24.882290 13059 raft_consensus.cc:2781] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:24.882458 13059 raft_consensus.cc:482] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:24.882578 13059 raft_consensus.cc:3037] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:24.885087 13059 raft_consensus.cc:504] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.885789 13059 leader_election.cc:288] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [CANDIDATE]: Term 1 election: Requested vote from peers 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643), 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:24.886142 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e462d655244c9abcc136a0124ad95a" candidate_uuid: "575c717d52484db0950744e401d2f059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d33408078f84f10818ad8ff3dd6e755"
I20260929 15:58:24.886310 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e462d655244c9abcc136a0124ad95a" candidate_uuid: "575c717d52484db0950744e401d2f059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773db94dadc942cbb7e44016c155608c"
I20260929 15:58:24.886363 12971 raft_consensus.cc:3037] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:24.886677 12830 raft_consensus.cc:3037] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:24.889657 12971 raft_consensus.cc:2454] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575c717d52484db0950744e401d2f059 in term 1.
I20260929 15:58:24.889932 12830 raft_consensus.cc:2454] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575c717d52484db0950744e401d2f059 in term 1.
I20260929 15:58:24.890100 12617 leader_election.cc:302] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d33408078f84f10818ad8ff3dd6e755, 575c717d52484db0950744e401d2f059; no voters:
I20260929 15:58:24.890406 13059 raft_consensus.cc:2781] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:24.890863 13059 raft_consensus.cc:686] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 1 LEADER]: Becoming Leader. State: Replica: 575c717d52484db0950744e401d2f059, State: Running, Role: LEADER
I20260929 15:58:24.891489 13059 consensus_queue.cc:237] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:24.897598 11755 catalog_manager.cc:5495] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 575c717d52484db0950744e401d2f059 (127.5.117.1). New cstate: current_term: 1 leader_uuid: "575c717d52484db0950744e401d2f059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } health_report { overall_health: UNKNOWN } } }
W20260929 15:58:24.919516 12730 tablet.cc:2369] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:24.949129 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.943037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.5.117.1:39409)
0929 15:58:24.943278 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.943284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.943303 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:58:24.943418 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.943422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.943527 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.943672 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:24.944040 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.944858 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.946348 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.946933 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.947031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.947609 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:24.948409 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:24.948562 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.948712 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.948787 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:58:24.959411 12830 raft_consensus.cc:1261] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Refusing update from remote peer 575c717d52484db0950744e401d2f059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:24.959436 12971 raft_consensus.cc:1261] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Refusing update from remote peer 575c717d52484db0950744e401d2f059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:24.960644 13064 consensus_queue.cc:1035] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:24.961058 13059 consensus_queue.cc:1035] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:25.007241 11755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:25.007712 11755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:25.016678 11755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff9b4e17af224cc69aca06fe48edd8b1: Sending DeleteTablet for 3 replicas of tablet 37e462d655244c9abcc136a0124ad95a
I20260929 15:58:25.018064 12662 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e462d655244c9abcc136a0124ad95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:25 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:53824
I20260929 15:58:25.018165 12951 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e462d655244c9abcc136a0124ad95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:25 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:33382
I20260929 15:58:25.019110 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e462d655244c9abcc136a0124ad95a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:25 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:44290
I20260929 15:58:25.029796 13079 tablet_replica.cc:330] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: stopping tablet replica
I20260929 15:58:25.031148 13079 raft_consensus.cc:2229] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:25.032691 13079 raft_consensus.cc:2258] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:25.035439 13077 tablet_replica.cc:330] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: stopping tablet replica
I20260929 15:58:25.036106 13077 raft_consensus.cc:2229] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:25.036549 13079 ts_tablet_manager.cc:1905] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:25.037061 13077 raft_consensus.cc:2258] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:25.038033 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 11724
I20260929 15:58:25.039243 11786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:25.046151 13077 ts_tablet_manager.cc:1905] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:25.047601 13078 tablet_replica.cc:330] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: stopping tablet replica
I20260929 15:58:25.048579 13078 raft_consensus.cc:2229] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:25.050457 13078 raft_consensus.cc:2258] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:25.050696 13079 ts_tablet_manager.cc:1918] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:25.050863 13079 log.cc:1196] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/37e462d655244c9abcc136a0124ad95a
I20260929 15:58:25.051680 13079 ts_tablet_manager.cc:1939] T 37e462d655244c9abcc136a0124ad95a P 773db94dadc942cbb7e44016c155608c: Deleting consensus metadata
I20260929 15:58:25.055666 13078 ts_tablet_manager.cc:1905] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:25.060729 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.020504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.5.117.61:38763)
0929 15:58:25.020758 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.020768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.036835 (+ 16067us) server_negotiation.cc:407] Connection header received
0929 15:58:25.037859 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.037864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.037957 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.038038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.041967 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.044931 (+ 2964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.047261 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.048037 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.048177 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.052319 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.052327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.052332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.052491 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.056784 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.057769 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.057775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.057779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.057943 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.058421 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.058425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.058428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.058862 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.059185 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.060023 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.060356 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 15:58:25.066946 13078 ts_tablet_manager.cc:1918] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:25.071694 13077 ts_tablet_manager.cc:1918] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:25.072011 13077 log.cc:1196] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/37e462d655244c9abcc136a0124ad95a
I20260929 15:58:25.072870 13077 ts_tablet_manager.cc:1939] T 37e462d655244c9abcc136a0124ad95a P 575c717d52484db0950744e401d2f059: Deleting consensus metadata
I20260929 15:58:25.075102 13078 log.cc:1196] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/37e462d655244c9abcc136a0124ad95a
I20260929 15:58:25.075667 13078 ts_tablet_manager.cc:1939] T 37e462d655244c9abcc136a0124ad95a P 1d33408078f84f10818ad8ff3dd6e755: Deleting consensus metadata
I20260929 15:58:25.081082 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.037330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.5.117.60:40857)
0929 15:58:25.037517 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.037523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.063271 (+ 25748us) server_negotiation.cc:407] Connection header received
0929 15:58:25.063435 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.063444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.063542 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.063666 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.065667 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.067002 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.068896 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.069598 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.069738 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.073818 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.073825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.073831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.073881 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.077874 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.078658 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.078664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.078668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.078832 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.079393 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.079398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.079408 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.079750 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.080043 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.080415 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.080656 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:58:25.220410 11742 catalog_manager.cc:4841] TS 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947): tablet 37e462d655244c9abcc136a0124ad95a (table test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]) successfully deleted
W20260929 15:58:25.220806 11742 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.117.2:36947, user_credentials={real_user=slave}} blocked reactor thread for 166965us
I20260929 15:58:25.221395 11742 catalog_manager.cc:4841] TS 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643): tablet 37e462d655244c9abcc136a0124ad95a (table test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]) successfully deleted
I20260929 15:58:25.221587 11742 catalog_manager.cc:4841] TS 575c717d52484db0950744e401d2f059 (127.5.117.1:39409): tablet 37e462d655244c9abcc136a0124ad95a (table test-table [id=18aa2eb08fce43fabe3f7a26f407bed6]) successfully deleted
I20260929 15:58:25.221897 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 11724
I20260929 15:58:25.275188 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.252826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.5.117.60:40857)
0929 15:58:25.253135 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.253145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.254924 (+ 1779us) server_negotiation.cc:407] Connection header received
0929 15:58:25.255296 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.255306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.255426 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.255527 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.258815 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.260521 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.263677 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.264441 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.264565 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.268090 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.268098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.268103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.268143 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.271505 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.272254 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.272260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.272264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.272398 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.272780 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.272788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.272792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.273091 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.273350 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.274361 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.274805 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:58:25.279727 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.253989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.5.117.61:38763)
0929 15:58:25.254262 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.254272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.263088 (+ 8816us) server_negotiation.cc:407] Connection header received
0929 15:58:25.263316 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.263322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.263411 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.263522 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.266281 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.267501 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.269430 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.270078 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.270217 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.273333 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.273338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.273342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.273382 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.276872 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.277657 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.277661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.277663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.277781 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.278209 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.278212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.278214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.278471 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.278689 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.279197 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.279375 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:58:25.991776 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.990811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:33291)
0929 15:58:25.991342 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:25.991503 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
W20260929 15:58:25.992332 12728 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:25.992762 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.991553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:58799)
0929 15:58:25.992364 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:25.992521 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":719,"threads_started":1}
W20260929 15:58:25.993119 13017 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:25.998701 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.997603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:39089)
0929 15:58:25.998162 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:25.998494 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":277,"threads_started":1}
W20260929 15:58:25.999143 12876 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39627 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.516599 13092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff9b4e17af224cc69aca06fe48edd8b1)
I20260929 15:58:26.516669 13093 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff9b4e17af224cc69aca06fe48edd8b1)
I20260929 15:58:26.516916 13092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } }
I20260929 15:58:26.516924 13093 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } }
I20260929 15:58:26.518461 13093 leader_election.cc:288] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627), 7f69514419264b078027313a5e037933 (127.5.117.61:38763)
I20260929 15:58:26.518471 13092 leader_election.cc:288] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627), f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60:40857)
I20260929 15:58:26.519923 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.518600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48142)
0929 15:58:26.519200 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.519437 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":293,"threads_started":1}
W20260929 15:58:26.521682 12533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.523064 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.518579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48136)
0929 15:58:26.522457 (+ 3878us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.522717 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3718,"thread_start_us":3548,"threads_started":1}
W20260929 15:58:26.529311 12343 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.529850 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.529287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48158)
0929 15:58:26.529450 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.529614 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 15:58:26.530164 12533 leader_election.cc:334] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627): Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.532169 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.531516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48162)
0929 15:58:26.531719 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.531898 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 15:58:26.532454 12343 leader_election.cc:334] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627): Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.532985 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.521901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:40857 (local address 127.0.0.1:36876)
0929 15:58:26.523472 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.523495 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.523745 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.524592 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.524601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.524643 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.525046 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.525063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.527307 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.527316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.530640 (+ 3324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.530666 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.532040 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.532070 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.532295 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.532342 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.532458 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.532633 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1443,"mutex_wait_us":1,"thread_start_us":6994,"threads_started":1}
I20260929 15:58:26.533062 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.519192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.5.117.60:40857)
0929 15:58:26.521959 (+ 2767us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.521966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.523616 (+ 1650us) server_negotiation.cc:407] Connection header received
0929 15:58:26.523928 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.523937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.524035 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.524152 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.525302 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.527069 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.530881 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.531858 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.532016 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.532158 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.532521 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.532749 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":2577,"thread_start_us":2447,"threads_started":1}
I20260929 15:58:26.533658 12558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f69514419264b078027313a5e037933" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" is_pre_election: true
I20260929 15:58:26.533983 12558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f69514419264b078027313a5e037933 in term 1.
I20260929 15:58:26.534515 12342 leader_election.cc:302] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f69514419264b078027313a5e037933, f6ef8ba1e54a4949a69fb8b18158ca44; no voters: ff9b4e17af224cc69aca06fe48edd8b1
I20260929 15:58:26.534956 13092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:58:26.535117 13092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff9b4e17af224cc69aca06fe48edd8b1)
I20260929 15:58:26.535274 13092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:26.536468 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.519251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:38763 (local address 127.0.0.1:50686)
0929 15:58:26.520364 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.520388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.520759 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.529843 (+ 9084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.529849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.529897 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.530290 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.530306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.532283 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.532288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.534041 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.534054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.535705 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.535710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.535895 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.535918 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.536023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.536185 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":2073,"threads_started":1}
I20260929 15:58:26.536639 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.522400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.5.117.61:38763)
0929 15:58:26.529436 (+ 7036us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.529442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.529527 (+ 85us) server_negotiation.cc:407] Connection header received
0929 15:58:26.529594 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.529599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.529675 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.530454 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.530784 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.532149 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.534226 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.535515 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.535674 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.535786 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.536078 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.536240 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6912,"thread_start_us":6666,"threads_started":1}
I20260929 15:58:26.537302 12367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f69514419264b078027313a5e037933" is_pre_election: true
I20260929 15:58:26.538266 13092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } }
I20260929 15:58:26.538682 12367 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6ef8ba1e54a4949a69fb8b18158ca44 in current term 2: Already voted for candidate 7f69514419264b078027313a5e037933 in this term.
I20260929 15:58:26.539000 13092 leader_election.cc:288] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 election: Requested vote from peers ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627), f6ef8ba1e54a4949a69fb8b18158ca44 (127.5.117.60:40857)
I20260929 15:58:26.539507 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.539051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48176)
0929 15:58:26.539247 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.539354 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 15:58:26.539425 12558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f69514419264b078027313a5e037933" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44"
I20260929 15:58:26.539659 12558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:26.539754 12533 leader_election.cc:302] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6ef8ba1e54a4949a69fb8b18158ca44; no voters: 7f69514419264b078027313a5e037933, ff9b4e17af224cc69aca06fe48edd8b1
I20260929 15:58:26.540632 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.540198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48182)
0929 15:58:26.540458 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.540541 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260929 15:58:26.540798 12343 leader_election.cc:334] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627): Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
I20260929 15:58:26.542212 12558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f69514419264b078027313a5e037933 in term 2.
I20260929 15:58:26.542436 13093 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:58:26.542639 12342 leader_election.cc:302] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f69514419264b078027313a5e037933, f6ef8ba1e54a4949a69fb8b18158ca44; no voters: ff9b4e17af224cc69aca06fe48edd8b1
I20260929 15:58:26.542932 13092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:58:26.543469 13092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [term 2 LEADER]: Becoming Leader. State: Replica: 7f69514419264b078027313a5e037933, State: Running, Role: LEADER
I20260929 15:58:26.543874 13092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } }
I20260929 15:58:26.545593 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f69514419264b078027313a5e037933. Latest consensus state: current_term: 2 leader_uuid: "7f69514419264b078027313a5e037933" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:26.545974 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:26.546757 13104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:26.550727 13104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:26.551479 13104 catalog_manager.cc:1259] Loaded cluster ID: c2d993f77c1649f4894ceb5b6d42d84c
I20260929 15:58:26.551558 13104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:26.552670 13104 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:26.553489 13104 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Loaded TSK: 0
I20260929 15:58:26.554329 13104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:26.574862 12356 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:26.583808 12356 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:26.588901 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.588318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48196)
0929 15:58:26.588550 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.588686 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 15:58:26.589082 12558 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [term 2 FOLLOWER]: Refusing update from remote peer 7f69514419264b078027313a5e037933: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 15:58:26.590237 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.589716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48198)
0929 15:58:26.589935 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.590038 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 15:58:26.590076 13092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 15:58:26.590431 12343 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 -> Peer ff9b4e17af224cc69aca06fe48edd8b1 (127.5.117.62:39627): Couldn't send request to peer ff9b4e17af224cc69aca06fe48edd8b1. Status: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:26.593149 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f69514419264b078027313a5e037933. Latest consensus state: current_term: 2 leader_uuid: "7f69514419264b078027313a5e037933" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:26.593449 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:26.595175 13106 mvcc.cc:205] Tried to move back new op lower bound from 7334696986979880960 to 7334696986807390208. Current Snapshot: MvccSnapshot[applied={T|T < 7334696986979880960}]
I20260929 15:58:26.595180 13092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7f69514419264b078027313a5e037933" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:26.595607 13092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:26.595696 13093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7f69514419264b078027313a5e037933" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ff9b4e17af224cc69aca06fe48edd8b1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39627 } } peers { permanent_uuid: "7f69514419264b078027313a5e037933" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 38763 } attrs { promote: false } } peers { permanent_uuid: "f6ef8ba1e54a4949a69fb8b18158ca44" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 40857 } attrs { promote: false } } }
I20260929 15:58:26.595963 13093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ef8ba1e54a4949a69fb8b18158ca44 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:26.632239 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.608915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34643 (local address 127.0.0.1:33396)
0929 15:58:26.609663 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.609681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.609817 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.624005 (+ 14188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.624013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.624070 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.624338 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.624355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.626228 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.626233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.628204 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.628217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.630873 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.630879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.631566 (+ 687us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.631606 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.631682 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.631829 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606}
I20260929 15:58:26.633147 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.609102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.5.117.3:34643)
0929 15:58:26.623392 (+ 14290us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.623402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.623430 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:58:26.623550 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.623560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.623686 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.624522 (+ 836us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.624786 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.625939 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.628931 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.630638 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.632225 (+ 1587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.632579 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.632638 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.632808 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14108,"thread_start_us":14694,"threads_started":1}
I20260929 15:58:26.634160 12951 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 (DEFAULT_TABLE table=test-table [id=e5a4352a84d54e22860ef6b405ec3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:26.634706 12951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2ece1b93884b66b4b0072f0a54c310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:26.636438 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.609965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36947 (local address 127.0.0.1:44300)
0929 15:58:26.610523 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.610546 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.610657 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.627484 (+ 16827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.627492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.627816 (+ 324us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.628254 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.628265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.629745 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.629751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.634257 (+ 4506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.634264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.635641 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.635647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.635826 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.635848 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.635954 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.636100 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":2793,"threads_started":1}
I20260929 15:58:26.636518 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.611190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.5.117.2:36947)
0929 15:58:26.627126 (+ 15936us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.627132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.627154 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:26.627249 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.627254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.627335 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.627444 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.628439 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.629582 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.634460 (+ 4878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.635421 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.635574 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.635697 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.635984 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.636182 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15798,"thread_start_us":15695,"threads_started":1}
I20260929 15:58:26.637475 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 (DEFAULT_TABLE table=test-table [id=e5a4352a84d54e22860ef6b405ec3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:26.637871 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.609678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39409 (local address 127.0.0.1:53844)
0929 15:58:26.610465 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.610487 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.610619 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.626493 (+ 15874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.626498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.626525 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.628692 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.628713 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.630337 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.630342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.633882 (+ 3540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.633894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.637003 (+ 3109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.637009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.637158 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.637191 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.637257 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.637389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1737}
I20260929 15:58:26.637993 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2ece1b93884b66b4b0072f0a54c310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:26.639403 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.611239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.5.117.1:39409)
0929 15:58:26.626004 (+ 14765us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.626012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.626032 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:26.626123 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.626127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.626240 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.626628 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.628867 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.630172 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.635567 (+ 5395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.636766 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.638707 (+ 1941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.638825 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.638883 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.639052 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14625,"thread_start_us":15066,"threads_started":1}
I20260929 15:58:26.640278 12662 tablet_service.cc:1467] Processing CreateTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 (DEFAULT_TABLE table=test-table [id=e5a4352a84d54e22860ef6b405ec3173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:26.640762 12662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e2ece1b93884b66b4b0072f0a54c310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:26.641959 13115 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Bootstrap starting.
I20260929 15:58:26.644804 13115 tablet_bootstrap.cc:654] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:26.650091 13118 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Bootstrap starting.
I20260929 15:58:26.653266 13118 tablet_bootstrap.cc:654] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:26.655913 13117 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Bootstrap starting.
I20260929 15:58:26.656474 13115 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: No bootstrap required, opened a new log
I20260929 15:58:26.656710 13115 ts_tablet_manager.cc:1397] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260929 15:58:26.658159 13115 raft_consensus.cc:348] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.658474 13115 raft_consensus.cc:374] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:26.658568 13117 tablet_bootstrap.cc:654] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:26.658630 13115 raft_consensus.cc:729] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d33408078f84f10818ad8ff3dd6e755, State: Initialized, Role: FOLLOWER
I20260929 15:58:26.658934 13115 consensus_queue.cc:260] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.659849 13115 ts_tablet_manager.cc:1428] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:26.661835 13118 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: No bootstrap required, opened a new log
I20260929 15:58:26.661980 13118 ts_tablet_manager.cc:1397] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260929 15:58:26.662828 13118 raft_consensus.cc:348] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.663105 13118 raft_consensus.cc:374] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:26.663216 13118 raft_consensus.cc:729] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773db94dadc942cbb7e44016c155608c, State: Initialized, Role: FOLLOWER
I20260929 15:58:26.663496 13118 consensus_queue.cc:260] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.664801 13018 heartbeater.cc:503] Master 127.5.117.61:38763 was elected leader, sending a full tablet report...
I20260929 15:58:26.665892 13117 tablet_bootstrap.cc:492] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: No bootstrap required, opened a new log
I20260929 15:58:26.666039 13117 ts_tablet_manager.cc:1397] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260929 15:58:26.666115 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.660540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:43803)
0929 15:58:26.661158 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.665824 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":344,"threads_started":1}
I20260929 15:58:26.667026 13117 raft_consensus.cc:348] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.667316 13117 raft_consensus.cc:374] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:26.667434 13117 raft_consensus.cc:729] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575c717d52484db0950744e401d2f059, State: Initialized, Role: FOLLOWER
I20260929 15:58:26.667713 13117 consensus_queue.cc:260] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.668762 13117 ts_tablet_manager.cc:1428] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:26.672106 13118 ts_tablet_manager.cc:1428] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Time spent starting tablet: real 0.010s user 0.000s sys 0.005s
I20260929 15:58:26.674053 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.673358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:43397)
0929 15:58:26.673642 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.673822 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:58:26.677022 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.669493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:51329)
0929 15:58:26.676546 (+ 7053us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.676718 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6891,"thread_start_us":6772,"threads_started":1}
I20260929 15:58:26.678098 12729 heartbeater.cc:503] Master 127.5.117.61:38763 was elected leader, sending a full tablet report...
I20260929 15:58:26.678722 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.678185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:58883)
0929 15:58:26.678375 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.678464 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 15:58:26.685261 12877 heartbeater.cc:503] Master 127.5.117.61:38763 was elected leader, sending a full tablet report...
I20260929 15:58:26.685696 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.673358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:58435)
0929 15:58:26.685071 (+ 11713us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.685309 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11353,"thread_start_us":11039,"threads_started":1}
I20260929 15:58:26.825560 13120 raft_consensus.cc:482] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:26.825876 13120 raft_consensus.cc:504] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.827733 13120 leader_election.cc:288] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575c717d52484db0950744e401d2f059 (127.5.117.1:39409), 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:26.835374 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.828476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:56283 (local address 127.5.117.1:39409)
0929 15:58:26.828719 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.828726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.828747 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:58:26.828846 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.828854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.828957 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.829062 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.829647 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.830689 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.833258 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.834388 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.834547 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.834662 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.834930 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.835077 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:58:26.835418 13122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.828210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:39409 (local address 127.5.117.3:56283)
0929 15:58:26.828411 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.828437 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.828622 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.829123 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.829126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.829151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.829412 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.829441 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.830874 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.830879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.833077 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.833099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.834591 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.834597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.834763 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.834789 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.834901 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.835049 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":30}
I20260929 15:58:26.836539 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.828713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:48819 (local address 127.5.117.2:36947)
0929 15:58:26.828994 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.829000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.829346 (+ 346us) server_negotiation.cc:407] Connection header received
0929 15:58:26.829611 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.829618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.829702 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.829805 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:26.830281 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.831205 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.834074 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.835451 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.835574 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.835692 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.836181 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.836330 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:58:26.836674 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.828565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36947 (local address 127.5.117.3:48819)
0929 15:58:26.829225 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.829246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:26.829401 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:26.829837 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:26.829841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:26.829864 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:26.830117 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.830125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.831341 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.831347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.833904 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:26.833920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.835801 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.835807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:26.835993 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.836025 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:58:26.836146 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 15:58:26.836273 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":6,"thread_start_us":350,"threads_started":1}
I20260929 15:58:26.836827 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2ece1b93884b66b4b0072f0a54c310" candidate_uuid: "1d33408078f84f10818ad8ff3dd6e755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575c717d52484db0950744e401d2f059" is_pre_election: true
I20260929 15:58:26.837271 12682 raft_consensus.cc:2454] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d33408078f84f10818ad8ff3dd6e755 in term 0.
I20260929 15:58:26.837348 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2ece1b93884b66b4b0072f0a54c310" candidate_uuid: "1d33408078f84f10818ad8ff3dd6e755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773db94dadc942cbb7e44016c155608c" is_pre_election: true
I20260929 15:58:26.837673 12830 raft_consensus.cc:2454] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d33408078f84f10818ad8ff3dd6e755 in term 0.
I20260929 15:58:26.837899 12906 leader_election.cc:302] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d33408078f84f10818ad8ff3dd6e755, 575c717d52484db0950744e401d2f059; no voters:
I20260929 15:58:26.838312 13120 raft_consensus.cc:2781] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:26.838459 13120 raft_consensus.cc:482] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:26.838573 13120 raft_consensus.cc:3037] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:26.841032 13120 raft_consensus.cc:504] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.841619 13120 leader_election.cc:288] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [CANDIDATE]: Term 1 election: Requested vote from peers 575c717d52484db0950744e401d2f059 (127.5.117.1:39409), 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947)
I20260929 15:58:26.842206 12682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2ece1b93884b66b4b0072f0a54c310" candidate_uuid: "1d33408078f84f10818ad8ff3dd6e755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575c717d52484db0950744e401d2f059"
I20260929 15:58:26.842257 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e2ece1b93884b66b4b0072f0a54c310" candidate_uuid: "1d33408078f84f10818ad8ff3dd6e755" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773db94dadc942cbb7e44016c155608c"
I20260929 15:58:26.842404 12682 raft_consensus.cc:3037] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:26.842453 12830 raft_consensus.cc:3037] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:26.844846 12830 raft_consensus.cc:2454] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d33408078f84f10818ad8ff3dd6e755 in term 1.
I20260929 15:58:26.845247 12906 leader_election.cc:302] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d33408078f84f10818ad8ff3dd6e755, 773db94dadc942cbb7e44016c155608c; no voters:
I20260929 15:58:26.845597 13120 raft_consensus.cc:2781] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:26.845667 12682 raft_consensus.cc:2454] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d33408078f84f10818ad8ff3dd6e755 in term 1.
I20260929 15:58:26.846164 13120 raft_consensus.cc:686] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 1 LEADER]: Becoming Leader. State: Replica: 1d33408078f84f10818ad8ff3dd6e755, State: Running, Role: LEADER
I20260929 15:58:26.846966 13120 consensus_queue.cc:237] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } }
I20260929 15:58:26.847230 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.846861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:56537)
0929 15:58:26.847000 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.847113 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:58:26.851373 12356 catalog_manager.cc:5495] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "1d33408078f84f10818ad8ff3dd6e755" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d33408078f84f10818ad8ff3dd6e755" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:26.921784 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.916289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.5.117.3:34643)
0929 15:58:26.916573 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.916578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.916605 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:58:26.916802 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.916806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.916901 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.917041 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:26.917406 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.918166 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.919416 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.919841 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.919932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.920445 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:26.921114 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:26.921232 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.921341 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.921416 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 15:58:26.927307 12830 raft_consensus.cc:1261] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Refusing update from remote peer 1d33408078f84f10818ad8ff3dd6e755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:26.927601 12682 raft_consensus.cc:1261] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Refusing update from remote peer 1d33408078f84f10818ad8ff3dd6e755: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:26.928282 13128 consensus_queue.cc:1035] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773db94dadc942cbb7e44016c155608c" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:26.928886 13120 consensus_queue.cc:1035] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575c717d52484db0950744e401d2f059" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 39409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:26.932152 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.931541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:56177)
0929 15:58:26.931818 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.931920 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 15:58:26.938449 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.932375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:56859)
0929 15:58:26.932553 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.932679 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 15:58:26.943473 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.942731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:41715)
0929 15:58:26.943109 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.943227 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260929 15:58:26.945932 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.941052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.2:59949)
0929 15:58:26.945491 (+ 4439us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.945667 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4283}
I20260929 15:58:26.952294 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.951371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.3:41875)
0929 15:58:26.951760 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.951860 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260929 15:58:26.953280 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.952816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.5.117.1:47583)
0929 15:58:26.953008 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.953117 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 15:58:26.962050 12356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:26.962364 12356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:26.968951 12356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f69514419264b078027313a5e037933: Sending DeleteTablet for 3 replicas of tablet 9e2ece1b93884b66b4b0072f0a54c310
I20260929 15:58:26.970238 12951 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:26 UTC) from {username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:33396
I20260929 15:58:26.970760 12662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:26 UTC) from {username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:53844
I20260929 15:58:26.971266 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e2ece1b93884b66b4b0072f0a54c310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:26 UTC) from {username='oryx', principal='oryx/127.5.117.61@KRBTEST.COM'} at 127.0.0.1:44300
I20260929 15:58:26.972702 13142 tablet_replica.cc:330] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: stopping tablet replica
I20260929 15:58:26.973276 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12535
I20260929 15:58:26.973443 13141 tablet_replica.cc:330] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: stopping tablet replica
I20260929 15:58:26.973827 13142 raft_consensus.cc:2229] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:26.974203 13141 raft_consensus.cc:2229] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:26.974622 13141 raft_consensus.cc:2258] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:26.975131 13142 raft_consensus.cc:2258] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:26.975989 13140 tablet_replica.cc:330] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: stopping tablet replica
I20260929 15:58:26.976258 13140 raft_consensus.cc:2229] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:26.976943 13140 raft_consensus.cc:2258] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:26.977305 13142 ts_tablet_manager.cc:1905] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:26.977972 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.977393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48204)
0929 15:58:26.977639 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.977761 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 15:58:26.979341 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.978763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39627 (local address 127.0.0.1:48218)
0929 15:58:26.979100 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:26.979192 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260929 15:58:26.980307 13140 ts_tablet_manager.cc:1905] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:26.985152 13142 ts_tablet_manager.cc:1918] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:26.985297 13142 log.cc:1196] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/9e2ece1b93884b66b4b0072f0a54c310
I20260929 15:58:26.986204 13142 ts_tablet_manager.cc:1939] T 9e2ece1b93884b66b4b0072f0a54c310 P 773db94dadc942cbb7e44016c155608c: Deleting consensus metadata
I20260929 15:58:26.986945 13141 ts_tablet_manager.cc:1905] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:26.987475 12343 catalog_manager.cc:4841] TS 773db94dadc942cbb7e44016c155608c (127.5.117.2:36947): tablet 9e2ece1b93884b66b4b0072f0a54c310 (table test-table [id=e5a4352a84d54e22860ef6b405ec3173]) successfully deleted
I20260929 15:58:26.989432 13140 ts_tablet_manager.cc:1918] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:26.989583 13140 log.cc:1196] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/9e2ece1b93884b66b4b0072f0a54c310
I20260929 15:58:26.991014 13140 ts_tablet_manager.cc:1939] T 9e2ece1b93884b66b4b0072f0a54c310 P 1d33408078f84f10818ad8ff3dd6e755: Deleting consensus metadata
I20260929 15:58:26.992345 12343 catalog_manager.cc:4841] TS 1d33408078f84f10818ad8ff3dd6e755 (127.5.117.3:34643): tablet 9e2ece1b93884b66b4b0072f0a54c310 (table test-table [id=e5a4352a84d54e22860ef6b405ec3173]) successfully deleted
I20260929 15:58:26.994827 13141 ts_tablet_manager.cc:1918] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:26.995020 13141 log.cc:1196] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/9e2ece1b93884b66b4b0072f0a54c310
I20260929 15:58:26.995600 13141 ts_tablet_manager.cc:1939] T 9e2ece1b93884b66b4b0072f0a54c310 P 575c717d52484db0950744e401d2f059: Deleting consensus metadata
I20260929 15:58:26.996929 12343 catalog_manager.cc:4841] TS 575c717d52484db0950744e401d2f059 (127.5.117.1:39409): tablet 9e2ece1b93884b66b4b0072f0a54c310 (table test-table [id=e5a4352a84d54e22860ef6b405ec3173]) successfully deleted
I20260929 15:58:27.000648 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.974150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.5.117.1:39409)
0929 15:58:26.974375 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.974382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.977245 (+ 2863us) server_negotiation.cc:407] Connection header received
0929 15:58:26.978599 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.978605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.978715 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.978825 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:26.985853 (+ 7028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.987035 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.989877 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.990313 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.990452 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.993763 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:26.993775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:26.993795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:26.993826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:26.997762 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.998413 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.998421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.998423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.998565 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.998953 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.998959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.998962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.999370 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:26.999724 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:27.000037 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:58:27.000298 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:58:27.001325 12722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:27.142948 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12535
I20260929 15:58:27.170787 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12739
I20260929 15:58:27.190938 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.172286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.5.117.2:36947)
0929 15:58:27.172514 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.172521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.172545 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:58:27.172872 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.172880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.172995 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.173102 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.175079 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.176278 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.178185 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.179046 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.179208 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.183238 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:27.183263 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:27.183280 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:27.183324 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:27.187007 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:27.188276 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:27.188285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:27.188287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:27.188468 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:27.189119 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:27.189123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:27.189126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:27.189571 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:27.189845 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:27.190330 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 15:58:27.190554 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:58:27.192283 12870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:27.329448 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12739
I20260929 15:58:27.350509 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12888
I20260929 15:58:27.366998 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.351429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.5.117.3:34643)
0929 15:58:27.351691 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.351698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.351923 (+ 225us) server_negotiation.cc:407] Connection header received
0929 15:58:27.352157 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.352164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.352271 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.352360 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.354406 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.355339 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.356777 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.357307 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.357452 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.360305 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:27.360318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:27.360342 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:27.360379 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:27.363678 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:27.364834 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:27.364842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:27.364845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:27.364977 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:27.365383 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:27.365387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:27.365389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:27.365778 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:27.366122 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:27.366412 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 15:58:27.366646 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 15:58:27.367810 13011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:27.515159 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12888
I20260929 15:58:27.538908 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12219
I20260929 15:58:27.540343 12387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:27.682710 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12219
I20260929 15:58:27.706591 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 12317
I20260929 15:58:27.707796 12578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:27.832698 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 12317
2026-09-29T15:58:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10637 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:27 dist-test-slave-b69v krb5kdc[13152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:27 dist-test-slave-b69v krb5kdc[13152](info): set up 2 sockets
Sep 29 15:58:27 dist-test-slave-b69v krb5kdc[13152](info): commencing operation
krb5kdc: starting...
W20260929 15:58:28.925287 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:29 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab.
Entry for principal oryx/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab.
I20260929 15:58:29.127696 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40583
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab
--principal=oryx/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:40583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:29.484169 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:29.484476 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:29.484650 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:29.492404 13165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:29.492555 13165 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:58:29.492602 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:29.492707 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:29.492811 13165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:29.492931 13165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:29.503927 13165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:40583
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.62.keytab
--principal=oryx/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:29.505685 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:29.507256 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:29.520081 13171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:29.520120 13170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:29.520035 13173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:29.521461 13165 server_base.cc:1034] running on GCE node
I20260929 15:58:29.522543 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:29.524019 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:29.525432 13165 hybrid_clock.cc:648] HybridClock initialized: now 1790697509525377 us; error 44 us; skew 500 ppm
I20260929 15:58:29.525813 13165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:29 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697509, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:29.529408 13165 init.cc:377] Logged in from keytab as oryx/127.5.117.62@KRBTEST.COM (short username oryx)
I20260929 15:58:29.531023 13165 webserver.cc:460] Webserver started at http://127.5.117.62:42865/ using document root <none> and password file <none>
I20260929 15:58:29.531774 13165 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:29.531909 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:29.532325 13165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:29.535508 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "a666f49218f74501819a35383cdb5482"
format_stamp: "Formatted at 2026-09-29 15:58:29 on dist-test-slave-b69v"
server_key: "b9d26ba5430abd85a29eab8b73132277"
server_key_iv: "77d17625fc6f263c2682a9102692d21b"
server_key_version: "encryptionkey@0"
I20260929 15:58:29.536404 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "a666f49218f74501819a35383cdb5482"
format_stamp: "Formatted at 2026-09-29 15:58:29 on dist-test-slave-b69v"
server_key: "b9d26ba5430abd85a29eab8b73132277"
server_key_iv: "77d17625fc6f263c2682a9102692d21b"
server_key_version: "encryptionkey@0"
I20260929 15:58:29.541733 13165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:58:29.545898 13180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:29.547390 13165 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:29.547611 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "a666f49218f74501819a35383cdb5482"
format_stamp: "Formatted at 2026-09-29 15:58:29 on dist-test-slave-b69v"
server_key: "b9d26ba5430abd85a29eab8b73132277"
server_key_iv: "77d17625fc6f263c2682a9102692d21b"
server_key_version: "encryptionkey@0"
I20260929 15:58:29.547863 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:29.580296 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:29.586292 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:29.586833 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:29.613430 13165 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:40583
I20260929 15:58:29.613458 13241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:40583 every 8 connection(s)
I20260929 15:58:29.615016 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:29.617081 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13165
I20260929 15:58:29.617475 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:29.618515 5588 external_mini_cluster.cc:1428] Setting key 93f8418f692097af88b481a15939085d
I20260929 15:58:29.621214 13242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:58:29 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:29.636642 13242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Bootstrap starting.
I20260929 15:58:29.640582 13242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:29.641870 13242 log.cc:824] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:29.645503 13242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: No bootstrap required, opened a new log
I20260929 15:58:29.646387 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:29.621311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.5.117.62:40583)
0929 15:58:29.622167 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:29.622180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:29.622341 (+ 161us) server_negotiation.cc:407] Connection header received
0929 15:58:29.624932 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:29.624955 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:29.625552 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:29.626039 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:29.628119 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.630285 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.631602 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.632701 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.632899 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.637536 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:29.637547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:29.637567 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:29.637610 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:29.641491 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:29.642722 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:29.642730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:29.642739 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:29.642988 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:29.643367 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:29.643373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:29.643376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:29.643944 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:29.644260 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:29.644692 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:58:29.645142 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":392,"threads_started":1}
I20260929 15:58:29.654075 13242 raft_consensus.cc:348] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } }
I20260929 15:58:29.654507 13242 raft_consensus.cc:374] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:29.654624 13242 raft_consensus.cc:729] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a666f49218f74501819a35383cdb5482, State: Initialized, Role: FOLLOWER
I20260929 15:58:29.655316 13242 consensus_queue.cc:260] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } }
I20260929 15:58:29.655648 13242 raft_consensus.cc:388] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:29.655815 13242 raft_consensus.cc:482] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:29.656031 13242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:29.659052 13242 raft_consensus.cc:504] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } }
I20260929 15:58:29.659657 13242 leader_election.cc:302] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a666f49218f74501819a35383cdb5482; no voters:
I20260929 15:58:29.660568 13242 leader_election.cc:288] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:29.660964 13247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:29.662041 13247 raft_consensus.cc:686] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [term 1 LEADER]: Becoming Leader. State: Replica: a666f49218f74501819a35383cdb5482, State: Running, Role: LEADER
I20260929 15:58:29.662619 13247 consensus_queue.cc:237] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } }
I20260929 15:58:29.663095 13242 sys_catalog.cc:564] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:29.666683 13248 sys_catalog.cc:455] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a666f49218f74501819a35383cdb5482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } } }
I20260929 15:58:29.666677 13249 sys_catalog.cc:455] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a666f49218f74501819a35383cdb5482. Latest consensus state: current_term: 1 leader_uuid: "a666f49218f74501819a35383cdb5482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a666f49218f74501819a35383cdb5482" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40583 } } }
I20260929 15:58:29.667222 13248 sys_catalog.cc:458] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:29.667246 13249 sys_catalog.cc:458] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:29.668525 13254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:29.676702 13254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:29.688367 13254 catalog_manager.cc:1347] Generated new cluster ID: f5bc1af479fa44e7ae326c97f606e7ea
I20260929 15:58:29.688530 13254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:29.698256 13254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:29.699697 13254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:29.709311 13254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Generated new TSK 0
I20260929 15:58:29.710290 13254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
I20260929 15:58:29.784192 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40583
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:30.160341 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:30.160696 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:30.160871 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:30.168272 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:30.168430 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:30.168569 13268 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:30.168704 13268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:30.179008 13268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:30.180670 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:30.182183 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:30.195602 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:30.195988 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:30.195763 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:30.197010 13268 server_base.cc:1034] running on GCE node
I20260929 15:58:30.197718 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:30.198841 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:30.200179 13268 hybrid_clock.cc:648] HybridClock initialized: now 1790697510200081 us; error 60 us; skew 500 ppm
I20260929 15:58:30.200547 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:30 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697510, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:30.204934 13268 init.cc:377] Logged in from keytab as oryx/127.5.117.1@KRBTEST.COM (short username oryx)
I20260929 15:58:30.207497 13268 webserver.cc:460] Webserver started at http://127.5.117.1:39151/ using document root <none> and password file <none>
I20260929 15:58:30.208393 13268 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:30.208487 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:30.208817 13268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:30.212486 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "08df1fe400154ddf85da1c41bde60727"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "ecee9b54365ef86f42984704968ce66a"
server_key_iv: "7796da0f8e3843290949a7c6941748dd"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.213402 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "08df1fe400154ddf85da1c41bde60727"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "ecee9b54365ef86f42984704968ce66a"
server_key_iv: "7796da0f8e3843290949a7c6941748dd"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.219070 13268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:30.222801 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:30.224411 13268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:58:30.224607 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "08df1fe400154ddf85da1c41bde60727"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "ecee9b54365ef86f42984704968ce66a"
server_key_iv: "7796da0f8e3843290949a7c6941748dd"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.224856 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:30.237396 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:30.241963 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:30.242265 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:30.245301 13268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:30.245421 13268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:30.245604 13268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:30.245702 13268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:58:30 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697510, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:30.274552 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.246043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:42687 (local address 127.5.117.62:40583)
0929 15:58:30.246689 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.246697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.247882 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 15:58:30.250234 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.250245 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.250377 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.250516 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.253317 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.254727 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.257672 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.258509 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.258691 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.265475 (+ 6784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:30.265575 (+ 100us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:30.265588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:30.265647 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:30.270547 (+ 4900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.271567 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.271575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.271583 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.271720 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.272259 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.272266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.272271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.272721 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:30.272878 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:30.273892 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0929 15:58:30.274138 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":351,"threads_started":1}
I20260929 15:58:30.276176 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.246688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.1:42687)
0929 15:58:30.247667 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.247722 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.249912 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.250806 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.250820 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.251756 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.252928 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.253015 (+ 87us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.254932 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.254941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.257508 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.257521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.259113 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.259121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.259803 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.261207 (+ 1404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.261235 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.265161 (+ 3926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:30.270711 (+ 5550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.270718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.270733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.271380 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.271841 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.271846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.271850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.272123 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.272844 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:30.272853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:30.273430 (+ 577us) client_negotiation.cc:769] Sending connection context
0929 15:58:30.273777 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 15:58:30.274409 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":432,"threads_started":1}
I20260929 15:58:30.294926 13268 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36873
I20260929 15:58:30.294958 13403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36873 every 8 connection(s)
I20260929 15:58:30.296612 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:30.299350 13404 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40583
I20260929 15:58:30.299659 13404 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:30.300511 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.284260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:57163 (local address 127.5.117.62:40583)
0929 15:58:30.284464 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.284470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.285398 (+ 928us) server_negotiation.cc:407] Connection header received
0929 15:58:30.285679 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.285683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.285755 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.285864 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.287357 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.288558 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.290949 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.291610 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.291765 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.293906 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:30.293914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:30.293923 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:30.293970 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:30.296913 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.297784 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.297792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.297798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.297970 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.298507 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.298515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.298519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.298998 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:30.299220 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:30.299765 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 15:58:30.300105 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:58:30.300625 13404 heartbeater.cc:511] Master 127.5.117.62:40583 requested a full tablet report, sending...
I20260929 15:58:30.300760 13382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.284455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.1:57163)
0929 15:58:30.285288 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.285318 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.285499 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.285902 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.285907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.286582 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.287204 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.287217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.288700 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.288705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.290757 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.290772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.291814 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.291820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.291994 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.292826 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.292849 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.293688 (+ 839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:30.297049 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.297057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.297061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.297599 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.298106 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.298112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.298115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.298284 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.299169 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:30.299179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:30.299455 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 15:58:30.299662 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 15:58:30.300138 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":472,"threads_started":1}
I20260929 15:58:30.303102 13196 ts_manager.cc:194] Registered new tserver with Master: 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873)
I20260929 15:58:30.303506 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13268
I20260929 15:58:30.303732 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:30.304715 5588 external_mini_cluster.cc:1428] Setting key c6c4b17e1c74d24568b26d2ebca6cc40
I20260929 15:58:30.304826 13196 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:42687
WARNING: no policy specified for oryx/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
I20260929 15:58:30.365669 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40583
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:30.729894 13408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:30.730176 13408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:30.730331 13408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:30.737601 13408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:30.737699 13408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:30.737829 13408 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:30.737964 13408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:30.747825 13408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:30.749478 13408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:30.751062 13408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:30.767302 13414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:30.767506 13413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:30.767848 13416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:30.768249 13408 server_base.cc:1034] running on GCE node
I20260929 15:58:30.769002 13408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:30.770239 13408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:30.771530 13408 hybrid_clock.cc:648] HybridClock initialized: now 1790697510771450 us; error 75 us; skew 500 ppm
I20260929 15:58:30.771879 13408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:30 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697510, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:30.775478 13408 init.cc:377] Logged in from keytab as oryx/127.5.117.2@KRBTEST.COM (short username oryx)
I20260929 15:58:30.777077 13408 webserver.cc:460] Webserver started at http://127.5.117.2:46705/ using document root <none> and password file <none>
I20260929 15:58:30.777945 13408 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:30.778038 13408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:30.778388 13408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:30.781826 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "0680b333153349cbbb6604efd4e60b96"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "118d74b5cf1d0fb5d6b844b67270cb7d"
server_key_iv: "b0cf1253e0d721cb234501b52d98b19f"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.782676 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "0680b333153349cbbb6604efd4e60b96"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "118d74b5cf1d0fb5d6b844b67270cb7d"
server_key_iv: "b0cf1253e0d721cb234501b52d98b19f"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.787904 13408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:58:30.791599 13423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:30.793092 13408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:30.793231 13408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "0680b333153349cbbb6604efd4e60b96"
format_stamp: "Formatted at 2026-09-29 15:58:30 on dist-test-slave-b69v"
server_key: "118d74b5cf1d0fb5d6b844b67270cb7d"
server_key_iv: "b0cf1253e0d721cb234501b52d98b19f"
server_key_version: "encryptionkey@0"
I20260929 15:58:30.793421 13408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:30.807399 13408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:30.811731 13408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:30.811967 13408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:30.815140 13408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:30.815246 13408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:30.815400 13408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:30.815485 13408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:58:30 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697510, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:30.839965 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.816388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:37925 (local address 127.5.117.62:40583)
0929 15:58:30.816904 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.816909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.818111 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 15:58:30.820406 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.820414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.820515 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.820675 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.822967 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.824022 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.826826 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.827444 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.827542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.833190 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:30.833195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:30.833198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:30.833230 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:30.835953 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.836829 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.836836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.836841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.836957 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.837567 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.837572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.837575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.837992 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:30.838172 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:30.839068 (+ 896us) server_negotiation.cc:299] Negotiation successful
0929 15:58:30.839378 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260929 15:58:30.841533 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.816966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.2:37925)
0929 15:58:30.817915 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.817962 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.820005 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.820927 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.820961 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.821908 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.822770 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.822792 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.824275 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.824284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.826578 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.826633 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.827735 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.827745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.828145 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.829038 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.829081 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.832920 (+ 3839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:30.836116 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.836124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.836140 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.836729 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.837126 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.837131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.837137 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.837401 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.838158 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:30.838170 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:30.838521 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 15:58:30.838929 (+ 408us) client_negotiation.cc:240] Negotiation successful
0929 15:58:30.839586 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":468,"threads_started":1}
I20260929 15:58:30.869500 13432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.854846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:37985 (local address 127.5.117.62:40583)
0929 15:58:30.855079 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.855087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.856300 (+ 1213us) server_negotiation.cc:407] Connection header received
0929 15:58:30.856584 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.856593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.856685 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.856770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.858309 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.859230 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.861451 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.862081 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.862215 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.864130 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:30.864136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:30.864141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:30.864181 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:30.866771 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.867394 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.867399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.867403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.867521 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:30.867881 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:30.867887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:30.867890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:30.868289 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:30.868474 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:30.868863 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:58:30.869136 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:58:30.869613 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.855018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.2:37985)
0929 15:58:30.856149 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.856180 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.856366 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.857030 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.857037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.857715 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.858152 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.858166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.859375 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.859383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.861315 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.861336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.862280 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.862288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.862457 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.863194 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.863221 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.863915 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:30.866903 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.866908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.866912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.867276 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.867630 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:30.867636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:30.867639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:30.867760 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:30.868413 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:30.868420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:30.868612 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 15:58:30.868753 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 15:58:30.869128 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":717,"threads_started":1}
I20260929 15:58:30.878969 13408 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:39747
I20260929 15:58:30.879009 13543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:39747 every 8 connection(s)
I20260929 15:58:30.880546 13408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:30.882678 13544 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40583
I20260929 15:58:30.882912 13544 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:30.883615 13544 heartbeater.cc:511] Master 127.5.117.62:40583 requested a full tablet report, sending...
I20260929 15:58:30.885046 13196 ts_manager.cc:194] Registered new tserver with Master: 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747)
I20260929 15:58:30.885870 13196 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:37925
I20260929 15:58:30.889782 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13408
I20260929 15:58:30.889935 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:30.890414 5588 external_mini_cluster.cc:1428] Setting key 3ba75e9fe537259ffc926e9c585ae157
WARNING: no policy specified for oryx/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
I20260929 15:58:30.945206 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40583
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:37521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:31.300998 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:31.301308 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:31.301501 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 15:58:31.307973 13404 heartbeater.cc:503] Master 127.5.117.62:40583 was elected leader, sending a full tablet report...
W20260929 15:58:31.308877 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:31.309010 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:31.309146 13548 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:31.309271 13548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:31.318495 13548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:37521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.117.62:40583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:31.319708 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:31.321163 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:31.334914 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:31.334883 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:31.335150 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:31.335790 13548 server_base.cc:1034] running on GCE node
I20260929 15:58:31.336544 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:31.337491 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:31.338742 13548 hybrid_clock.cc:648] HybridClock initialized: now 1790697511338671 us; error 46 us; skew 500 ppm
I20260929 15:58:31.339061 13548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697511, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:31.343140 13548 init.cc:377] Logged in from keytab as oryx/127.5.117.3@KRBTEST.COM (short username oryx)
I20260929 15:58:31.345211 13548 webserver.cc:460] Webserver started at http://127.5.117.3:45567/ using document root <none> and password file <none>
I20260929 15:58:31.346113 13548 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:31.346235 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:31.346627 13548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:31.350131 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "cf0d2cc7de334901a8288347ef22e294"
format_stamp: "Formatted at 2026-09-29 15:58:31 on dist-test-slave-b69v"
server_key: "45d761ae76f0019a291fe1e1a754f2c5"
server_key_iv: "1d73077f6524b257ebd6974e032b3034"
server_key_version: "encryptionkey@0"
I20260929 15:58:31.351078 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "cf0d2cc7de334901a8288347ef22e294"
format_stamp: "Formatted at 2026-09-29 15:58:31 on dist-test-slave-b69v"
server_key: "45d761ae76f0019a291fe1e1a754f2c5"
server_key_iv: "1d73077f6524b257ebd6974e032b3034"
server_key_version: "encryptionkey@0"
I20260929 15:58:31.357227 13548 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:58:31.361300 13563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:31.362875 13548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 15:58:31.363059 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "cf0d2cc7de334901a8288347ef22e294"
format_stamp: "Formatted at 2026-09-29 15:58:31 on dist-test-slave-b69v"
server_key: "45d761ae76f0019a291fe1e1a754f2c5"
server_key_iv: "1d73077f6524b257ebd6974e032b3034"
server_key_version: "encryptionkey@0"
I20260929 15:58:31.363317 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:31.385759 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:31.390393 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:31.390725 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:31.394042 13548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:31.394212 13548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:31.394412 13548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:31.394476 13548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697511, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:31.424073 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.394463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:50505 (local address 127.5.117.62:40583)
0929 15:58:31.395084 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.395092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.396491 (+ 1399us) server_negotiation.cc:407] Connection header received
0929 15:58:31.399195 (+ 2704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.399201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.399310 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.399444 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.402056 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.403564 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.406351 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.407226 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.407389 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.413688 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:31.413698 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:31.413706 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:31.413770 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:31.418674 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.420147 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.420151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.420153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.420298 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.421014 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.421019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.421022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.421517 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:31.421706 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.423106 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.423444 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":280,"threads_started":1}
I20260929 15:58:31.426625 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.395245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.3:50505)
0929 15:58:31.396283 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.396344 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.398847 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.399763 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.399779 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.400757 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:31.401772 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.401806 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.403796 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.403802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.406137 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.406150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.407608 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.407620 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.408001 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.409147 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:31.409222 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:31.413261 (+ 4039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:31.418884 (+ 5623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:31.418920 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:31.418953 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:31.419944 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:31.420430 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:31.420437 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:31.420444 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:31.420855 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:31.421688 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:31.421697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:31.422417 (+ 720us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.422937 (+ 520us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.423847 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":432,"threads_started":1}
I20260929 15:58:31.443370 13548 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:40651
I20260929 15:58:31.443393 13683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:40651 every 8 connection(s)
I20260929 15:58:31.445106 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:31.448114 13684 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40583
I20260929 15:58:31.448402 13684 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:31.448908 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13548
I20260929 15:58:31.449120 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:31.449402 13684 heartbeater.cc:511] Master 127.5.117.62:40583 requested a full tablet report, sending...
I20260929 15:58:31.449694 5588 external_mini_cluster.cc:1428] Setting key 6ffd4b845cda2bb00335cbcb8d7ed8ef
I20260929 15:58:31.451520 13197 ts_manager.cc:194] Registered new tserver with Master: cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:31.453241 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:58:31.453377 13197 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:50505
I20260929 15:58:31.454389 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.435326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.5.117.3:47091)
0929 15:58:31.436759 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.436792 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.437025 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.437518 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.437528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.438488 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:31.439158 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.439172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.440708 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.440713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.442270 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.442285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.443642 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.443649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.443809 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.444651 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:31.444671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:31.445691 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:31.450265 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:31.450270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:31.450275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:31.450931 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:31.451797 (+ 866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:31.451819 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:31.451824 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:31.452187 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:31.453098 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:31.453105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:31.453359 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.453516 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.453898 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":917,"threads_started":1}
I20260929 15:58:31.455361 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.435382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:47091 (local address 127.5.117.62:40583)
0929 15:58:31.435668 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.435680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.436917 (+ 1237us) server_negotiation.cc:407] Connection header received
0929 15:58:31.437223 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.437231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.437335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.437460 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.439320 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.440550 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.442500 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.443183 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.443326 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.445981 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:31.445987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:31.445992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:31.446034 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:31.450090 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.451099 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.451105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.451108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.451274 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.452496 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.452501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.452508 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.452965 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:31.453131 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.454749 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.455009 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:31.525379 13571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:31.513893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.5.117.62:40583)
0929 15:58:31.514202 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.514207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.514303 (+ 96us) server_negotiation.cc:407] Connection header received
0929 15:58:31.514484 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.514489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.514583 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.514677 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.516277 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.517090 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.518405 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.518868 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.518967 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.525108 (+ 6141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 15:58:31 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697511, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:31.545218 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.529581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.5.117.62:40583)
0929 15:58:31.529853 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.529859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.530205 (+ 346us) server_negotiation.cc:407] Connection header received
0929 15:58:31.530353 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.530358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.530429 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.530518 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.531922 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.532794 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.534076 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.534559 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.534672 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.538995 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:31.539003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:31.539007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:31.539067 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:31.542120 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.543027 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.543032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.543035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.543194 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:31.543759 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:31.543763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:31.543765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:31.544179 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:31.544408 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.544721 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.544932 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:58:31.550730 13197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:31.558647 13197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:31.598865 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.573745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39747 (local address 127.0.0.1:45720)
0929 15:58:31.586816 (+ 13071us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.586873 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.587098 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.588296 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.588307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.588338 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:31.588792 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.588807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.591244 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.591253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.594424 (+ 3171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.594446 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.597450 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.597459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.597742 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.597813 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.598016 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.598295 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12710,"thread_start_us":14145,"threads_started":1}
I20260929 15:58:31.600282 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.573920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.5.117.2:39747)
0929 15:58:31.587080 (+ 13160us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.587089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.587126 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:58:31.587454 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.587478 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.587871 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.588161 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:31.589031 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.591014 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.594705 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.596725 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.596990 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.597295 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.599561 (+ 2266us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.599841 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12971,"thread_start_us":12835,"threads_started":1}
I20260929 15:58:31.601882 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.573826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36873 (local address 127.0.0.1:52926)
0929 15:58:31.588225 (+ 14399us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.588245 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.588439 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.589373 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.589381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.589415 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:31.590000 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.590015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.593751 (+ 3736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.593759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.596753 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.596772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.600965 (+ 4193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.600974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.601239 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.601272 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.601418 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.601566 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14188,"thread_start_us":13368,"threads_started":1}
I20260929 15:58:31.602003 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.574200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.5.117.1:36873)
0929 15:58:31.586358 (+ 12158us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.586372 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.588335 (+ 1963us) server_negotiation.cc:407] Connection header received
0929 15:58:31.588641 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.588651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.588873 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.589068 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:31.590227 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.592282 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.597033 (+ 4751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.599064 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.599158 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.599339 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.601489 (+ 2150us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.601673 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11891,"thread_start_us":12150,"threads_started":1}
I20260929 15:58:31.605598 13474 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a (DEFAULT_TABLE table=test-table [id=86d9af3385ad46d99e2cca42495f00a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:31.605597 13334 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a (DEFAULT_TABLE table=test-table [id=86d9af3385ad46d99e2cca42495f00a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:31.607097 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcdd9c8c98141cc87a16442c35b322a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:31.607120 13474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcdd9c8c98141cc87a16442c35b322a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:31.608879 13702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.588932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:40651 (local address 127.0.0.1:32976)
0929 15:58:31.599189 (+ 10257us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.599211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.599350 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.601826 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.601832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.601856 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:31.602037 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.602050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.604453 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.604467 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.606239 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.606256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.608164 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.608170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.608356 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.608387 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.608501 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.608633 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10138,"thread_start_us":12224,"threads_started":1}
I20260929 15:58:31.609288 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.589169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.5.117.3:40651)
0929 15:58:31.601161 (+ 11992us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.601171 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.601209 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:58:31.601324 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.601332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.601609 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.602217 (+ 608us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:31.602605 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.604298 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.606432 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.607975 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.608142 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.608373 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.608547 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.608731 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11789,"thread_start_us":11572,"threads_started":1}
I20260929 15:58:31.613049 13614 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a (DEFAULT_TABLE table=test-table [id=86d9af3385ad46d99e2cca42495f00a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:31.614415 13614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcdd9c8c98141cc87a16442c35b322a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:31.633165 13708 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Bootstrap starting.
I20260929 15:58:31.636121 13709 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Bootstrap starting.
I20260929 15:58:31.637861 13708 tablet_bootstrap.cc:654] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:31.638801 13710 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Bootstrap starting.
I20260929 15:58:31.639392 13708 log.cc:824] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:31.639439 13709 tablet_bootstrap.cc:654] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:31.640673 13709 log.cc:824] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:31.643404 13710 tablet_bootstrap.cc:654] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:31.644872 13710 log.cc:824] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:31.647374 13708 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: No bootstrap required, opened a new log
I20260929 15:58:31.647373 13709 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: No bootstrap required, opened a new log
I20260929 15:58:31.647543 13710 tablet_bootstrap.cc:492] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: No bootstrap required, opened a new log
I20260929 15:58:31.647667 13709 ts_tablet_manager.cc:1397] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260929 15:58:31.647667 13708 ts_tablet_manager.cc:1397] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260929 15:58:31.647845 13710 ts_tablet_manager.cc:1397] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 15:58:31.655443 13710 raft_consensus.cc:348] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.655849 13710 raft_consensus.cc:374] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:31.655956 13710 raft_consensus.cc:729] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Initialized, Role: FOLLOWER
I20260929 15:58:31.657009 13710 consensus_queue.cc:260] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.658236 13684 heartbeater.cc:503] Master 127.5.117.62:40583 was elected leader, sending a full tablet report...
I20260929 15:58:31.658928 13710 ts_tablet_manager.cc:1428] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 15:58:31.658807 13708 raft_consensus.cc:348] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.658807 13709 raft_consensus.cc:348] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.659224 13708 raft_consensus.cc:374] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:31.659224 13709 raft_consensus.cc:374] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:31.659322 13709 raft_consensus.cc:729] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Initialized, Role: FOLLOWER
I20260929 15:58:31.659322 13708 raft_consensus.cc:729] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Initialized, Role: FOLLOWER
I20260929 15:58:31.659986 13708 consensus_queue.cc:260] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.659934 13709 consensus_queue.cc:260] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.662117 13709 ts_tablet_manager.cc:1428] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 15:58:31.662281 13544 heartbeater.cc:503] Master 127.5.117.62:40583 was elected leader, sending a full tablet report...
I20260929 15:58:31.663298 13708 ts_tablet_manager.cc:1428] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260929 15:58:31.697331 13685 tablet.cc:2369] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:31.799942 13405 tablet.cc:2369] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:31.883005 13545 tablet.cc:2369] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:31.933187 13716 raft_consensus.cc:482] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:31.933702 13716 raft_consensus.cc:504] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.935804 13716 leader_election.cc:288] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:31.945487 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.936652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:46829 (local address 127.5.117.2:39747)
0929 15:58:31.936961 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.936969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.937272 (+ 303us) server_negotiation.cc:407] Connection header received
0929 15:58:31.937599 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.937621 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.937732 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.937844 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:31.938590 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.939740 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.942781 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.944140 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.944296 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.944422 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.945144 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.945265 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 15:58:31.945561 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.937516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45933 (local address 127.5.117.3:40651)
0929 15:58:31.937765 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.937774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.938117 (+ 343us) server_negotiation.cc:407] Connection header received
0929 15:58:31.938275 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.938279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.938344 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.938423 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:31.939002 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.940027 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.942781 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.944158 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.944296 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.944422 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:31.945118 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 15:58:31.945275 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:58:31.945674 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.936447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39747 (local address 127.5.117.1:46829)
0929 15:58:31.937136 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.937154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.937376 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.937869 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.937875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.937903 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:31.938383 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.938417 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.939906 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.939910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.942554 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.942571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.944653 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.944657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.944859 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.944938 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.945083 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.945223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":105,"thread_start_us":352,"threads_started":1}
I20260929 15:58:31.946257 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.937312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:40651 (local address 127.5.117.1:45933)
0929 15:58:31.937980 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.937998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.938146 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.938462 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.938469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.938502 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:31.938849 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.938858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.940215 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.940221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.942554 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.942571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.944623 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.944636 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.944855 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.944933 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 15:58:31.945763 (+ 830us) client_negotiation.cc:240] Negotiation successful
0929 15:58:31.945914 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":78,"thread_start_us":352,"threads_started":1}
I20260929 15:58:31.946908 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcdd9c8c98141cc87a16442c35b322a" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96" is_pre_election: true
I20260929 15:58:31.947327 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcdd9c8c98141cc87a16442c35b322a" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294" is_pre_election: true
I20260929 15:58:31.947494 13495 raft_consensus.cc:2454] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 0.
I20260929 15:58:31.947988 13634 raft_consensus.cc:2454] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 0.
I20260929 15:58:31.948328 13284 leader_election.cc:302] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727; no voters:
I20260929 15:58:31.948820 13716 raft_consensus.cc:2781] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:31.948993 13716 raft_consensus.cc:482] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:31.949162 13716 raft_consensus.cc:3037] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:31.951699 13716 raft_consensus.cc:504] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.952438 13716 leader_election.cc:288] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 election: Requested vote from peers 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:31.953020 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcdd9c8c98141cc87a16442c35b322a" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96"
I20260929 15:58:31.953174 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcdd9c8c98141cc87a16442c35b322a" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294"
I20260929 15:58:31.953320 13495 raft_consensus.cc:3037] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:31.953436 13634 raft_consensus.cc:3037] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:31.955828 13495 raft_consensus.cc:2454] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 1.
I20260929 15:58:31.956053 13634 raft_consensus.cc:2454] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 1.
I20260929 15:58:31.956326 13284 leader_election.cc:302] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727; no voters:
I20260929 15:58:31.956660 13716 raft_consensus.cc:2781] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:31.957146 13716 raft_consensus.cc:686] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 1 LEADER]: Becoming Leader. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Running, Role: LEADER
I20260929 15:58:31.957692 13716 consensus_queue.cc:237] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:31.963153 13195 catalog_manager.cc:5495] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08df1fe400154ddf85da1c41bde60727 (127.5.117.1). New cstate: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:32.050580 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.044660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.5.117.1:36873)
0929 15:58:32.044885 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.044891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.044980 (+ 89us) server_negotiation.cc:407] Connection header received
0929 15:58:32.045276 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.045281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.045377 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.045589 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:32.045980 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.046746 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.048191 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.048618 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.048761 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.049216 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.049911 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.050100 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.050166 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.050258 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:58:32.063119 13634 raft_consensus.cc:1261] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Refusing update from remote peer 08df1fe400154ddf85da1c41bde60727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:32.063946 13495 raft_consensus.cc:1261] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Refusing update from remote peer 08df1fe400154ddf85da1c41bde60727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:32.064376 13719 consensus_queue.cc:1035] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.064829 13716 consensus_queue.cc:1035] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.080241 13724 mvcc.cc:205] Tried to move back new op lower bound from 7334697009396363264 to 7334697008982822912. Current Snapshot: MvccSnapshot[applied={T|T < 7334697009396363264}]
I20260929 15:58:32.089805 13196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:32.090147 13196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:32.094748 13196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Sending DeleteTablet for 3 replicas of tablet 0fcdd9c8c98141cc87a16442c35b322a
I20260929 15:58:32.095956 13614 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:32976
I20260929 15:58:32.096093 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 15:58:32.096021 13474 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260929 15:58:32.098481 13613 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fcdd9c8c98141cc87a16442c35b322a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:32976
I20260929 15:58:32.101883 13195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:32.103030 13195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:32.106777 13731 tablet_replica.cc:330] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: stopping tablet replica
I20260929 15:58:32.107388 13731 raft_consensus.cc:2229] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:32.108291 13731 raft_consensus.cc:2258] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.113867 13731 ts_tablet_manager.cc:1905] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.114842 13473 tablet_service.cc:1467] Processing CreateTablet for tablet 0703d6430b5543cab8765b6b05ffd342 (DEFAULT_TABLE table=test-table [id=4ed74e7664df41b5887856877fe09c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:32.115608 13473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703d6430b5543cab8765b6b05ffd342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.118582 13732 tablet_replica.cc:330] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: stopping tablet replica
I20260929 15:58:32.119246 13732 raft_consensus.cc:2229] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:32.119935 13732 raft_consensus.cc:2258] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.120944 13613 tablet_service.cc:1467] Processing CreateTablet for tablet 0703d6430b5543cab8765b6b05ffd342 (DEFAULT_TABLE table=test-table [id=4ed74e7664df41b5887856877fe09c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:32.121562 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703d6430b5543cab8765b6b05ffd342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.125207 13334 tablet_service.cc:1467] Processing CreateTablet for tablet 0703d6430b5543cab8765b6b05ffd342 (DEFAULT_TABLE table=test-table [id=4ed74e7664df41b5887856877fe09c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:32.125820 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0703d6430b5543cab8765b6b05ffd342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.127071 13731 ts_tablet_manager.cc:1918] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:32.127295 13731 log.cc:1196] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/0fcdd9c8c98141cc87a16442c35b322a
I20260929 15:58:32.127985 13731 ts_tablet_manager.cc:1939] T 0fcdd9c8c98141cc87a16442c35b322a P 08df1fe400154ddf85da1c41bde60727: Deleting consensus metadata
I20260929 15:58:32.132010 13732 ts_tablet_manager.cc:1905] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.139909 13732 ts_tablet_manager.cc:1918] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:32.141655 13182 catalog_manager.cc:4841] TS 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873): tablet 0fcdd9c8c98141cc87a16442c35b322a (table test-table [id=86d9af3385ad46d99e2cca42495f00a9]) successfully deleted
I20260929 15:58:32.142225 13732 log.cc:1196] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/0fcdd9c8c98141cc87a16442c35b322a
I20260929 15:58:32.142899 13732 ts_tablet_manager.cc:1939] T 0fcdd9c8c98141cc87a16442c35b322a P cf0d2cc7de334901a8288347ef22e294: Deleting consensus metadata
I20260929 15:58:32.145252 13181 catalog_manager.cc:4841] TS cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651): tablet 0fcdd9c8c98141cc87a16442c35b322a (table test-table [id=86d9af3385ad46d99e2cca42495f00a9]) successfully deleted
W20260929 15:58:32.145949 13181 catalog_manager.cc:4808] TS cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651): delete failed for tablet 0fcdd9c8c98141cc87a16442c35b322a because the tablet was not found. No further retry: Not found: Tablet not found: 0fcdd9c8c98141cc87a16442c35b322a
I20260929 15:58:32.150379 13709 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Bootstrap starting.
I20260929 15:58:32.150602 13710 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Bootstrap starting.
I20260929 15:58:32.150949 13733 tablet_replica.cc:330] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: stopping tablet replica
I20260929 15:58:32.151590 13733 raft_consensus.cc:2229] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:32.152285 13709 tablet_bootstrap.cc:654] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.152397 13733 raft_consensus.cc:2258] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.153214 13710 tablet_bootstrap.cc:654] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.155256 13708 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Bootstrap starting.
I20260929 15:58:32.156121 13710 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: No bootstrap required, opened a new log
I20260929 15:58:32.156320 13710 ts_tablet_manager.cc:1397] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 15:58:32.156946 13709 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: No bootstrap required, opened a new log
I20260929 15:58:32.157124 13709 ts_tablet_manager.cc:1397] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 15:58:32.157423 13708 tablet_bootstrap.cc:654] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.157337 13710 raft_consensus.cc:348] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.157631 13710 raft_consensus.cc:374] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.157745 13710 raft_consensus.cc:729] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.158142 13709 raft_consensus.cc:348] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.158404 13709 raft_consensus.cc:374] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.158820 13710 consensus_queue.cc:260] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.158541 13709 raft_consensus.cc:729] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.160218 13709 consensus_queue.cc:260] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.160617 13733 ts_tablet_manager.cc:1905] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.162447 13710 ts_tablet_manager.cc:1428] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 15:58:32.165817 13709 ts_tablet_manager.cc:1428] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20260929 15:58:32.168500 13708 tablet_bootstrap.cc:492] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: No bootstrap required, opened a new log
I20260929 15:58:32.168725 13708 ts_tablet_manager.cc:1397] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.001s
I20260929 15:58:32.169826 13708 raft_consensus.cc:348] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.170091 13708 raft_consensus.cc:374] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.170207 13708 raft_consensus.cc:729] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.170506 13708 consensus_queue.cc:260] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.171173 13708 ts_tablet_manager.cc:1428] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:58:32.174425 13715 raft_consensus.cc:482] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:32.174688 13715 raft_consensus.cc:504] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.176951 13715 leader_election.cc:288] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:32.179306 13733 ts_tablet_manager.cc:1918] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:32.179514 13733 log.cc:1196] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/0fcdd9c8c98141cc87a16442c35b322a
I20260929 15:58:32.180328 13733 ts_tablet_manager.cc:1939] T 0fcdd9c8c98141cc87a16442c35b322a P 0680b333153349cbbb6604efd4e60b96: Deleting consensus metadata
I20260929 15:58:32.182571 13181 catalog_manager.cc:4841] TS 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747): tablet 0fcdd9c8c98141cc87a16442c35b322a (table test-table [id=86d9af3385ad46d99e2cca42495f00a9]) successfully deleted
I20260929 15:58:32.190928 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.178092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:52509 (local address 127.5.117.1:36873)
0929 15:58:32.178322 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.178330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.183019 (+ 4689us) server_negotiation.cc:407] Connection header received
0929 15:58:32.183249 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.183258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.183366 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.183472 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:32.184365 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.185544 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.188114 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.189753 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.189905 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.190027 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.190394 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.190571 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:58:32.190994 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.177412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36873 (local address 127.5.117.2:52509)
0929 15:58:32.182871 (+ 5459us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.182898 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.183067 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.183689 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.183692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.183714 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:32.184134 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.184151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.185725 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.185730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.187895 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.187911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.189998 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.190004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.190175 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.190216 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.190345 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.190493 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5317,"mutex_wait_us":4,"thread_start_us":5930,"threads_started":1}
I20260929 15:58:32.191987 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.177464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:40651 (local address 127.5.117.2:36105)
0929 15:58:32.183641 (+ 6177us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.183659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.183796 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.184194 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.184202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.184227 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:32.184674 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.184686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.186096 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.186101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.188494 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.188509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.190850 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.190856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.191451 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.191498 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.191602 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.191738 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6020,"thread_start_us":5115,"threads_started":1}
I20260929 15:58:32.192148 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.178585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36105 (local address 127.5.117.3:40651)
0929 15:58:32.178801 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.178808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.183768 (+ 4960us) server_negotiation.cc:407] Connection header received
0929 15:58:32.183927 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.183933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.184075 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.184184 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:32.184848 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.185952 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.188820 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.190447 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.190577 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.190680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.191657 (+ 977us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.191820 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:58:32.192126 13354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0703d6430b5543cab8765b6b05ffd342" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727" is_pre_election: true
I20260929 15:58:32.192670 13354 raft_consensus.cc:2454] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 0.
I20260929 15:58:32.192965 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0703d6430b5543cab8765b6b05ffd342" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294" is_pre_election: true
I20260929 15:58:32.193284 13634 raft_consensus.cc:2454] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 0.
I20260929 15:58:32.193419 13425 leader_election.cc:302] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727; no voters:
I20260929 15:58:32.193843 13715 raft_consensus.cc:2781] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:32.193989 13715 raft_consensus.cc:482] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:32.194101 13715 raft_consensus.cc:3037] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.196444 13715 raft_consensus.cc:504] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.197182 13715 leader_election.cc:288] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 election: Requested vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:32.197553 13354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0703d6430b5543cab8765b6b05ffd342" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727"
I20260929 15:58:32.197799 13354 raft_consensus.cc:3037] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.197743 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0703d6430b5543cab8765b6b05ffd342" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294"
I20260929 15:58:32.197955 13634 raft_consensus.cc:3037] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.200831 13354 raft_consensus.cc:2454] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 1.
I20260929 15:58:32.201238 13425 leader_election.cc:302] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727; no voters:
I20260929 15:58:32.201300 13634 raft_consensus.cc:2454] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 1.
I20260929 15:58:32.201575 13715 raft_consensus.cc:2781] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:32.202167 13715 raft_consensus.cc:686] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 1 LEADER]: Becoming Leader. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Running, Role: LEADER
I20260929 15:58:32.202647 13715 consensus_queue.cc:237] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.206415 13195 catalog_manager.cc:5495] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0680b333153349cbbb6604efd4e60b96 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "0680b333153349cbbb6604efd4e60b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } health_report { overall_health: UNKNOWN } } }
Sep 29 15:58:32 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697509, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.62@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:32.263024 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.235055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40583 (local address 127.0.0.1:54314)
0929 15:58:32.236387 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.236409 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.236548 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.237031 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.237035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.237916 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:32.238192 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.238202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.239872 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.239877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.240928 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.240941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.241998 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.242002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.242147 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.252043 (+ 9896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:32.252121 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:32.255504 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:32.259944 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:32.259957 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:32.259964 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:32.260664 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:32.261096 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:32.261099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:32.261101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:32.261262 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:32.261937 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:32.261944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:32.262146 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.262299 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.262544 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"thread_start_us":1060,"threads_started":1}
I20260929 15:58:32.263127 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.235109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.5.117.62:40583)
0929 15:58:32.235889 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.235898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.236511 (+ 613us) server_negotiation.cc:407] Connection header received
0929 15:58:32.236688 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.236694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.236795 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.236956 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:32.238387 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.239722 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.241098 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.241802 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.241943 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.255840 (+ 13897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:32.255848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:32.255853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:32.255894 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:32.259784 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:32.260801 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:32.260807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:32.260811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:32.260979 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:32.261375 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:32.261380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:32.261385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:32.261779 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:32.261937 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.262450 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.262685 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260929 15:58:32.270038 13195 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54314:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:58:32.271431 13195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:32.281536 13474 tablet_service.cc:1467] Processing CreateTablet for tablet a871dcd975ee41f883baa3324af294e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edc70189887f499799baff0c7a0784da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:32.281538 13334 tablet_service.cc:1467] Processing CreateTablet for tablet a871dcd975ee41f883baa3324af294e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edc70189887f499799baff0c7a0784da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:32.281731 13613 tablet_service.cc:1467] Processing CreateTablet for tablet a871dcd975ee41f883baa3324af294e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edc70189887f499799baff0c7a0784da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:32.282349 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a871dcd975ee41f883baa3324af294e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.282346 13474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a871dcd975ee41f883baa3324af294e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.282409 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a871dcd975ee41f883baa3324af294e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:32.290966 13709 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: Bootstrap starting.
I20260929 15:58:32.291193 13710 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: Bootstrap starting.
I20260929 15:58:32.292759 13709 tablet_bootstrap.cc:654] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.293188 13710 tablet_bootstrap.cc:654] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.298264 13708 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: Bootstrap starting.
I20260929 15:58:32.300506 13708 tablet_bootstrap.cc:654] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:32.313700 13710 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: No bootstrap required, opened a new log
I20260929 15:58:32.313967 13710 ts_tablet_manager.cc:1397] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.015s
I20260929 15:58:32.315016 13710 raft_consensus.cc:348] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.315378 13710 raft_consensus.cc:374] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.315478 13710 raft_consensus.cc:729] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.316944 13710 consensus_queue.cc:260] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.317489 13708 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: No bootstrap required, opened a new log
I20260929 15:58:32.317741 13708 ts_tablet_manager.cc:1397] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20260929 15:58:32.317554 13714 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.318032 13714 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.319018 13708 raft_consensus.cc:348] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.319399 13708 raft_consensus.cc:374] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.319564 13708 raft_consensus.cc:729] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.320000 13708 consensus_queue.cc:260] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.321542 13719 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.321934 13719 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.322799 13710 ts_tablet_manager.cc:1428] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: Time spent starting tablet: real 0.009s user 0.002s sys 0.002s
I20260929 15:58:32.323266 13708 ts_tablet_manager.cc:1428] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 15:58:32.325232 13709 tablet_bootstrap.cc:492] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: No bootstrap required, opened a new log
I20260929 15:58:32.325564 13709 ts_tablet_manager.cc:1397] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.004s
I20260929 15:58:32.327378 13709 raft_consensus.cc:348] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.327780 13709 raft_consensus.cc:374] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:32.327894 13709 raft_consensus.cc:729] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Initialized, Role: FOLLOWER
I20260929 15:58:32.328289 13709 consensus_queue.cc:260] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.328871 13715 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.329252 13715 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.329264 13709 ts_tablet_manager.cc:1428] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:32.599249 13719 raft_consensus.cc:482] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:32.599613 13719 raft_consensus.cc:504] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.600162 13715 consensus_queue.cc:1035] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.600467 13719 leader_election.cc:288] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:32.601161 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a871dcd975ee41f883baa3324af294e8" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96" is_pre_election: true
I20260929 15:58:32.601233 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a871dcd975ee41f883baa3324af294e8" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294" is_pre_election: true
I20260929 15:58:32.601445 13495 raft_consensus.cc:2454] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 0.
I20260929 15:58:32.601543 13634 raft_consensus.cc:2454] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 0.
I20260929 15:58:32.602000 13284 leader_election.cc:302] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:32.602568 13719 raft_consensus.cc:2781] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:32.602705 13719 raft_consensus.cc:482] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:32.602854 13719 raft_consensus.cc:3037] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.605504 13719 raft_consensus.cc:504] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.606302 13719 leader_election.cc:288] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 election: Requested vote from peers 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:32.607347 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a871dcd975ee41f883baa3324af294e8" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294"
I20260929 15:58:32.607697 13634 raft_consensus.cc:3037] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.608888 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a871dcd975ee41f883baa3324af294e8" candidate_uuid: "08df1fe400154ddf85da1c41bde60727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96"
I20260929 15:58:32.609185 13495 raft_consensus.cc:3037] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:32.611081 13634 raft_consensus.cc:2454] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 1.
I20260929 15:58:32.612207 13284 leader_election.cc:302] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:32.612552 13719 raft_consensus.cc:2781] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:32.612766 13719 raft_consensus.cc:686] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [term 1 LEADER]: Becoming Leader. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Running, Role: LEADER
I20260929 15:58:32.613191 13719 consensus_queue.cc:237] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:32.618855 13495 raft_consensus.cc:2454] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08df1fe400154ddf85da1c41bde60727 in term 1.
I20260929 15:58:32.625885 13751 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: TxnStatusTablet state changed. Reason: New leader 08df1fe400154ddf85da1c41bde60727. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.626394 13751 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:32.627825 13195 catalog_manager.cc:5495] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08df1fe400154ddf85da1c41bde60727 (127.5.117.1). New cstate: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:32.624637 13715 consensus_queue.cc:1035] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.643193 13758 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:58:32.643714 13758 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:58:32.697399 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.690865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.5.117.1:36873)
0929 15:58:32.691105 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.691112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.691411 (+ 299us) server_negotiation.cc:407] Connection header received
0929 15:58:32.691796 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.691801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.691878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.691998 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:32.692528 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.693346 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.695342 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.695798 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.695905 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.696528 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.696678 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.696755 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.696940 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.697021 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 15:58:32.697461 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.690835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36873 (local address 127.0.0.1:53870)
0929 15:58:32.691112 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.691135 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.691548 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.692026 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.692031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.692044 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:32.692326 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.692339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.693504 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.693511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.695173 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.695189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.695977 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.695984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.696252 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.696333 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.696785 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.696807 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.696906 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.697108 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 15:58:32.702335 13495 raft_consensus.cc:1261] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Refusing update from remote peer 08df1fe400154ddf85da1c41bde60727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:32.702428 13634 raft_consensus.cc:1261] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Refusing update from remote peer 08df1fe400154ddf85da1c41bde60727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:32.703126 13751 consensus_queue.cc:1035] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.703476 13719 consensus_queue.cc:1035] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:32.709650 13714 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: TxnStatusTablet state changed. Reason: New leader 08df1fe400154ddf85da1c41bde60727. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.709990 13714 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.710739 13739 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: TxnStatusTablet state changed. Reason: New leader 08df1fe400154ddf85da1c41bde60727. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.711115 13739 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.714424 13719 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.714815 13719 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:32.716392 13714 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.716671 13714 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P cf0d2cc7de334901a8288347ef22e294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.716987 13719 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.717250 13719 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 08df1fe400154ddf85da1c41bde60727: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:32.722033 13739 tablet_replica.cc:439] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "08df1fe400154ddf85da1c41bde60727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } } }
I20260929 15:58:32.722301 13739 tablet_replica.cc:442] T a871dcd975ee41f883baa3324af294e8 P 0680b333153349cbbb6604efd4e60b96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:32.738579 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.727614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.5.117.2:39747)
0929 15:58:32.732509 (+ 4895us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.732514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.732530 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:58:32.732808 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.732814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.733003 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.733416 (+ 413us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:32.733916 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.734706 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.736161 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.736604 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.736752 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.737201 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.737899 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.738083 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.738147 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.738240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":4662,"threads_started":1}
I20260929 15:58:32.758422 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.751211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36873 (local address 127.5.117.2:35305)
0929 15:58:32.752124 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.752145 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.752332 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.752847 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.752852 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.752866 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:32.753200 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.753210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.754802 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.754806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.755963 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.755976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.757149 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.757155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.757292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.757368 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.757860 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.757870 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.757937 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.758077 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":447,"threads_started":1}
I20260929 15:58:32.758450 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.751241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35305 (local address 127.5.117.1:36873)
0929 15:58:32.751522 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.751530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.752241 (+ 711us) server_negotiation.cc:407] Connection header received
0929 15:58:32.752504 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.752511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.752617 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.752752 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:32.753374 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.754648 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.756189 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.756932 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.757086 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.757574 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.757758 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.757864 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.757963 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.758067 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:58:32.814770 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.806194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39747 (local address 127.5.117.1:38461)
0929 15:58:32.807117 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.807173 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.807350 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.807979 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.807984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.807999 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:32.808440 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.808449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.809937 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.809946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.811756 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.811771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.813076 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.813089 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.813371 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.813523 (+ 152us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.814166 (+ 643us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.814181 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 15:58:32.814268 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 15:58:32.814398 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":545,"threads_started":1}
I20260929 15:58:32.814749 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.806506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:38461 (local address 127.5.117.2:39747)
0929 15:58:32.806744 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.806755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.807270 (+ 515us) server_negotiation.cc:407] Connection header received
0929 15:58:32.807536 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.807540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.807653 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.807745 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:32.808763 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.809768 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.811914 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.812627 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.812773 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.813855 (+ 1082us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:32.814012 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:32.814120 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:32.814315 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 15:58:32.814415 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 15:58:32.865397 13195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:32.865743 13195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:32.869747 13195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Sending DeleteTablet for 3 replicas of tablet 0703d6430b5543cab8765b6b05ffd342
I20260929 15:58:32.870786 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 0703d6430b5543cab8765b6b05ffd342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 15:58:32.871378 13613 tablet_service.cc:1514] Processing DeleteTablet for tablet 0703d6430b5543cab8765b6b05ffd342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:32976
I20260929 15:58:32.871888 13474 tablet_service.cc:1514] Processing DeleteTablet for tablet 0703d6430b5543cab8765b6b05ffd342 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:32 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260929 15:58:32.887691 13778 tablet_replica.cc:330] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: stopping tablet replica
I20260929 15:58:32.888753 13778 raft_consensus.cc:2229] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:32.889603 13778 raft_consensus.cc:2258] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.896678 13780 tablet_replica.cc:330] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: stopping tablet replica
I20260929 15:58:32.896930 13780 raft_consensus.cc:2229] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:32.897696 13780 raft_consensus.cc:2258] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.898840 13780 ts_tablet_manager.cc:1905] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.911005 13779 tablet_replica.cc:330] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: stopping tablet replica
I20260929 15:58:32.911314 13779 raft_consensus.cc:2229] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:32.911648 13779 raft_consensus.cc:2258] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:32.913206 13779 ts_tablet_manager.cc:1905] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.920194 13780 ts_tablet_manager.cc:1918] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:32.920369 13780 log.cc:1196] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/0703d6430b5543cab8765b6b05ffd342
I20260929 15:58:32.920931 13780 ts_tablet_manager.cc:1939] T 0703d6430b5543cab8765b6b05ffd342 P 0680b333153349cbbb6604efd4e60b96: Deleting consensus metadata
I20260929 15:58:32.921241 13779 ts_tablet_manager.cc:1918] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:32.921377 13779 log.cc:1196] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/0703d6430b5543cab8765b6b05ffd342
I20260929 15:58:32.921872 13779 ts_tablet_manager.cc:1939] T 0703d6430b5543cab8765b6b05ffd342 P cf0d2cc7de334901a8288347ef22e294: Deleting consensus metadata
I20260929 15:58:32.922470 13181 catalog_manager.cc:4841] TS 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747): tablet 0703d6430b5543cab8765b6b05ffd342 (table test-table [id=4ed74e7664df41b5887856877fe09c04]) successfully deleted
I20260929 15:58:32.923014 13181 catalog_manager.cc:4841] TS cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651): tablet 0703d6430b5543cab8765b6b05ffd342 (table test-table [id=4ed74e7664df41b5887856877fe09c04]) successfully deleted
I20260929 15:58:32.930559 13778 ts_tablet_manager.cc:1905] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:32.961424 13778 ts_tablet_manager.cc:1918] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:32.961649 13778 log.cc:1196] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/0703d6430b5543cab8765b6b05ffd342
I20260929 15:58:32.962210 13778 ts_tablet_manager.cc:1939] T 0703d6430b5543cab8765b6b05ffd342 P 08df1fe400154ddf85da1c41bde60727: Deleting consensus metadata
I20260929 15:58:32.963732 13182 catalog_manager.cc:4841] TS 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873): tablet 0703d6430b5543cab8765b6b05ffd342 (table test-table [id=4ed74e7664df41b5887856877fe09c04]) successfully deleted
Sep 29 15:58:32 dist-test-slave-b69v krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260929 15:58:32.988746 5588 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.111s user 0.011s sys 0.013s
Sep 29 15:58:33 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:33 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:33.062567 13788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:33.022069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.5.117.62:40583)
0929 15:58:33.046248 (+ 24179us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.046258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.047416 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 15:58:33.047513 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.047521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.047652 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.047784 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.050066 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.051352 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.053839 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.054520 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.054701 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.062155 (+ 7454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54328: BlockingRecv error: recv got EOF from 127.0.0.1:54328 (error 108)
Metrics: {"server-negotiator.queue_time_us":23966,"thread_start_us":24214,"threads_started":1}
Sep 29 15:58:33 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM
I20260929 15:58:33.129698 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.097098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.5.117.62:40583)
0929 15:58:33.097323 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.097330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.107943 (+ 10613us) server_negotiation.cc:407] Connection header received
0929 15:58:33.108608 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.108617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.108726 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.108816 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.111051 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.112363 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.114404 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.115096 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.115199 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.121358 (+ 6159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:33.121366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:33.121371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:33.121422 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:33.125868 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:33.126798 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:33.126804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:33.126808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:33.126989 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:33.127513 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:33.127520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:33.127523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:33.127996 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:33.128283 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.128962 (+ 679us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.129252 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:58:33.136220 13195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:33.137115 13195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:33.147087 13334 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d (DEFAULT_TABLE table=test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.147719 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7dfeafaf04dc8b5262b9060064c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.149389 13474 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d (DEFAULT_TABLE table=test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.149516 13613 tablet_service.cc:1467] Processing CreateTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d (DEFAULT_TABLE table=test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.149963 13474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7dfeafaf04dc8b5262b9060064c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.149988 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b7dfeafaf04dc8b5262b9060064c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.164391 13796 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Bootstrap starting.
I20260929 15:58:33.169549 13794 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Bootstrap starting.
I20260929 15:58:33.170094 13795 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Bootstrap starting.
I20260929 15:58:33.172355 13794 tablet_bootstrap.cc:654] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.173502 13796 tablet_bootstrap.cc:654] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.174906 13795 tablet_bootstrap.cc:654] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.185945 13795 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: No bootstrap required, opened a new log
I20260929 15:58:33.188093 13795 ts_tablet_manager.cc:1397] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260929 15:58:33.189679 13794 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: No bootstrap required, opened a new log
I20260929 15:58:33.189849 13794 ts_tablet_manager.cc:1397] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20260929 15:58:33.191053 13794 raft_consensus.cc:348] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.191370 13794 raft_consensus.cc:374] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.191289 13796 tablet_bootstrap.cc:492] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: No bootstrap required, opened a new log
I20260929 15:58:33.191466 13794 raft_consensus.cc:729] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.191710 13796 ts_tablet_manager.cc:1397] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.005s
I20260929 15:58:33.191787 13794 consensus_queue.cc:260] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.192901 13796 raft_consensus.cc:348] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.193295 13796 raft_consensus.cc:374] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.193457 13796 raft_consensus.cc:729] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.193810 13796 consensus_queue.cc:260] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.194429 13795 raft_consensus.cc:348] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.195379 13794 ts_tablet_manager.cc:1428] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:58:33.197381 13796 ts_tablet_manager.cc:1428] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 15:58:33.197679 13795 raft_consensus.cc:374] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.199191 13795 raft_consensus.cc:729] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.204319 13795 consensus_queue.cc:260] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.205143 13795 ts_tablet_manager.cc:1428] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20260929 15:58:33.254171 13714 raft_consensus.cc:482] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:33.254498 13714 raft_consensus.cc:504] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.256706 13714 leader_election.cc:288] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747)
I20260929 15:58:33.282526 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.258499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36411 (local address 127.5.117.1:36873)
0929 15:58:33.258718 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.258724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.273869 (+ 15145us) server_negotiation.cc:407] Connection header received
0929 15:58:33.274154 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.274162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.274267 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.274364 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:33.275261 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.276470 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.279596 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.280955 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.281085 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.281200 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.282031 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.282195 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 15:58:33.282541 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.258127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:37205 (local address 127.5.117.2:39747)
0929 15:58:33.258346 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.258353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.274643 (+ 16290us) server_negotiation.cc:407] Connection header received
0929 15:58:33.274950 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.274955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.275045 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.275127 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:33.275886 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.277006 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.279605 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.280965 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.281075 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.281200 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.282031 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.282195 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:58:33.283466 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.257283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36873 (local address 127.5.117.3:36411)
0929 15:58:33.273728 (+ 16445us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:33.273760 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:33.273975 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:33.274561 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:33.274571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:33.274598 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:33.275050 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.275062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.276655 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.276660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.279413 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.279427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.281471 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.281478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.281753 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.281865 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 15:58:33.282886 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0929 15:58:33.283021 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16239,"mutex_wait_us":118,"spinlock_wait_cycles":2432,"thread_start_us":21192,"threads_started":1}
I20260929 15:58:33.283883 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.257662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39747 (local address 127.5.117.3:37205)
0929 15:58:33.274526 (+ 16864us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:33.274544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:33.274768 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:33.275345 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:33.275349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:33.275373 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:33.275719 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.275729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.277183 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.277189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.279412 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.279427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.281414 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.281423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.281756 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.281872 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 15:58:33.283104 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0929 15:58:33.283241 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16743,"mutex_wait_us":191,"thread_start_us":15710,"threads_started":1}
I20260929 15:58:33.284219 13353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7dfeafaf04dc8b5262b9060064c2d" candidate_uuid: "cf0d2cc7de334901a8288347ef22e294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727" is_pre_election: true
I20260929 15:58:33.284617 13353 raft_consensus.cc:2454] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0d2cc7de334901a8288347ef22e294 in term 0.
I20260929 15:58:33.285320 13565 leader_election.cc:302] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:33.285530 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7dfeafaf04dc8b5262b9060064c2d" candidate_uuid: "cf0d2cc7de334901a8288347ef22e294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96" is_pre_election: true
I20260929 15:58:33.285758 13714 raft_consensus.cc:2781] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:33.285876 13714 raft_consensus.cc:482] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:33.285887 13495 raft_consensus.cc:2454] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf0d2cc7de334901a8288347ef22e294 in term 0.
I20260929 15:58:33.285964 13714 raft_consensus.cc:3037] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.288727 13714 raft_consensus.cc:504] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.289343 13714 leader_election.cc:288] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [CANDIDATE]: Term 1 election: Requested vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747)
I20260929 15:58:33.291800 13495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7dfeafaf04dc8b5262b9060064c2d" candidate_uuid: "cf0d2cc7de334901a8288347ef22e294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0680b333153349cbbb6604efd4e60b96"
I20260929 15:58:33.292099 13495 raft_consensus.cc:3037] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.292364 13353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b7dfeafaf04dc8b5262b9060064c2d" candidate_uuid: "cf0d2cc7de334901a8288347ef22e294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727"
I20260929 15:58:33.292594 13353 raft_consensus.cc:3037] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.294951 13353 raft_consensus.cc:2454] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0d2cc7de334901a8288347ef22e294 in term 1.
I20260929 15:58:33.295348 13565 leader_election.cc:302] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:33.295650 13714 raft_consensus.cc:2781] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:33.296008 13495 raft_consensus.cc:2454] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf0d2cc7de334901a8288347ef22e294 in term 1.
I20260929 15:58:33.296325 13714 raft_consensus.cc:686] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 1 LEADER]: Becoming Leader. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Running, Role: LEADER
I20260929 15:58:33.296839 13714 consensus_queue.cc:237] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.301014 13197 catalog_manager.cc:5495] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf0d2cc7de334901a8288347ef22e294 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "cf0d2cc7de334901a8288347ef22e294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } health_report { overall_health: HEALTHY } } }
I20260929 15:58:33.340850 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.326856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.5.117.3:40651)
0929 15:58:33.334962 (+ 8106us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.334967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.334990 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:58:33.335094 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.335099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.335196 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.335398 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:33.335945 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.336836 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.338213 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.338645 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.338784 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.339296 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:33.339984 (+ 688us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:33.340180 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.340310 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.340414 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7975,"thread_start_us":7816,"threads_started":1}
I20260929 15:58:33.345310 13353 raft_consensus.cc:1261] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Refusing update from remote peer cf0d2cc7de334901a8288347ef22e294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:33.345983 13495 raft_consensus.cc:1261] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Refusing update from remote peer cf0d2cc7de334901a8288347ef22e294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:33.346611 13802 consensus_queue.cc:1035] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:33.348150 13802 consensus_queue.cc:1035] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:33.372771 13760 mvcc.cc:205] Tried to move back new op lower bound from 7334697014654017536 to 7334697014468624384. Current Snapshot: MvccSnapshot[applied={T|T < 7334697014654017536}]
I20260929 15:58:33.389757 13806 mvcc.cc:205] Tried to move back new op lower bound from 7334697014654017536 to 7334697014468624384. Current Snapshot: MvccSnapshot[applied={T|T < 7334697014654017536}]
I20260929 15:58:33.395741 13197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:33.396093 13197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:33.401752 13197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Sending DeleteTablet for 3 replicas of tablet 09b7dfeafaf04dc8b5262b9060064c2d
I20260929 15:58:33.404878 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 15:58:33.405262 13778 tablet_replica.cc:330] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: stopping tablet replica
I20260929 15:58:33.405601 13778 raft_consensus.cc:2229] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:33.405951 13778 raft_consensus.cc:2258] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.406512 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 15:58:33.407486 13474 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260929 15:58:33.407740 13778 ts_tablet_manager.cc:1905] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.407850 13780 tablet_replica.cc:330] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: stopping tablet replica
I20260929 15:58:33.408108 13780 raft_consensus.cc:2229] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:33.408392 13780 raft_consensus.cc:2258] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.408619 13613 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b7dfeafaf04dc8b5262b9060064c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:32976
I20260929 15:58:33.408929 13779 tablet_replica.cc:330] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: stopping tablet replica
I20260929 15:58:33.409171 13779 raft_consensus.cc:2229] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:33.409816 13779 raft_consensus.cc:2258] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.411119 13779 ts_tablet_manager.cc:1905] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.411126 13196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:33.411926 13196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:33.418869 13779 ts_tablet_manager.cc:1918] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:33.419075 13779 log.cc:1196] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/09b7dfeafaf04dc8b5262b9060064c2d
I20260929 15:58:33.419584 13779 ts_tablet_manager.cc:1939] T 09b7dfeafaf04dc8b5262b9060064c2d P cf0d2cc7de334901a8288347ef22e294: Deleting consensus metadata
I20260929 15:58:33.421599 13613 tablet_service.cc:1467] Processing CreateTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc (DEFAULT_TABLE table=test-table [id=9166227f64534bb7bb6750a5ed3860d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.422140 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f5b522ed814f00a7d9bacfc73631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.424540 13474 tablet_service.cc:1467] Processing CreateTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc (DEFAULT_TABLE table=test-table [id=9166227f64534bb7bb6750a5ed3860d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.425101 13474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f5b522ed814f00a7d9bacfc73631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.429303 13334 tablet_service.cc:1467] Processing CreateTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc (DEFAULT_TABLE table=test-table [id=9166227f64534bb7bb6750a5ed3860d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:33.429524 13778 ts_tablet_manager.cc:1918] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:33.429757 13778 log.cc:1196] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/09b7dfeafaf04dc8b5262b9060064c2d
I20260929 15:58:33.429920 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79f5b522ed814f00a7d9bacfc73631fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:33.430351 13778 ts_tablet_manager.cc:1939] T 09b7dfeafaf04dc8b5262b9060064c2d P 08df1fe400154ddf85da1c41bde60727: Deleting consensus metadata
I20260929 15:58:33.432610 13181 catalog_manager.cc:4841] TS cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651): tablet 09b7dfeafaf04dc8b5262b9060064c2d (table test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]) successfully deleted
I20260929 15:58:33.433303 13182 catalog_manager.cc:4841] TS 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873): tablet 09b7dfeafaf04dc8b5262b9060064c2d (table test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]) successfully deleted
W20260929 15:58:33.433617 13182 catalog_manager.cc:4808] TS 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873): delete failed for tablet 09b7dfeafaf04dc8b5262b9060064c2d because the tablet was not found. No further retry: Not found: Tablet not found: 09b7dfeafaf04dc8b5262b9060064c2d
I20260929 15:58:33.452176 13795 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Bootstrap starting.
I20260929 15:58:33.454540 13795 tablet_bootstrap.cc:654] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.465245 13795 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: No bootstrap required, opened a new log
I20260929 15:58:33.465474 13795 ts_tablet_manager.cc:1397] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260929 15:58:33.466470 13795 raft_consensus.cc:348] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.466763 13795 raft_consensus.cc:374] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.466897 13795 raft_consensus.cc:729] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.467144 13795 consensus_queue.cc:260] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.467706 13795 ts_tablet_manager.cc:1428] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:58:33.475072 13796 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Bootstrap starting.
I20260929 15:58:33.476854 13794 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Bootstrap starting.
I20260929 15:58:33.477336 13796 tablet_bootstrap.cc:654] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.479434 13794 tablet_bootstrap.cc:654] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:33.484956 13780 ts_tablet_manager.cc:1905] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.495996 13780 ts_tablet_manager.cc:1918] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:33.497413 13780 log.cc:1196] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/09b7dfeafaf04dc8b5262b9060064c2d
I20260929 15:58:33.497937 13780 ts_tablet_manager.cc:1939] T 09b7dfeafaf04dc8b5262b9060064c2d P 0680b333153349cbbb6604efd4e60b96: Deleting consensus metadata
I20260929 15:58:33.499228 13181 catalog_manager.cc:4841] TS 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747): tablet 09b7dfeafaf04dc8b5262b9060064c2d (table test-table [id=d16e234e6d5a4deca9ecf1361b9e321f]) successfully deleted
I20260929 15:58:33.501231 13794 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: No bootstrap required, opened a new log
I20260929 15:58:33.501426 13794 ts_tablet_manager.cc:1397] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20260929 15:58:33.502382 13794 raft_consensus.cc:348] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.502691 13794 raft_consensus.cc:374] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.502776 13794 raft_consensus.cc:729] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0d2cc7de334901a8288347ef22e294, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.503053 13794 consensus_queue.cc:260] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.503639 13794 ts_tablet_manager.cc:1428] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:58:33.505241 13796 tablet_bootstrap.cc:492] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: No bootstrap required, opened a new log
I20260929 15:58:33.505438 13796 ts_tablet_manager.cc:1397] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.005s
I20260929 15:58:33.506258 13796 raft_consensus.cc:348] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.506543 13796 raft_consensus.cc:374] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:33.506618 13796 raft_consensus.cc:729] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08df1fe400154ddf85da1c41bde60727, State: Initialized, Role: FOLLOWER
I20260929 15:58:33.506870 13796 consensus_queue.cc:260] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.507429 13796 ts_tablet_manager.cc:1428] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:58:33.754267 13739 raft_consensus.cc:482] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:33.754599 13739 raft_consensus.cc:504] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.755616 13739 leader_election.cc:288] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:33.756402 13353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f5b522ed814f00a7d9bacfc73631fc" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727" is_pre_election: true
I20260929 15:58:33.756389 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f5b522ed814f00a7d9bacfc73631fc" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294" is_pre_election: true
I20260929 15:58:33.756726 13353 raft_consensus.cc:2454] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 0.
I20260929 15:58:33.756763 13634 raft_consensus.cc:2454] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 0.
I20260929 15:58:33.757401 13424 leader_election.cc:302] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:33.757789 13739 raft_consensus.cc:2781] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:33.757941 13739 raft_consensus.cc:482] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:33.758056 13739 raft_consensus.cc:3037] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.760638 13739 raft_consensus.cc:504] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.761384 13739 leader_election.cc:288] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 election: Requested vote from peers 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873), cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651)
I20260929 15:58:33.761966 13353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f5b522ed814f00a7d9bacfc73631fc" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08df1fe400154ddf85da1c41bde60727"
I20260929 15:58:33.762044 13634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79f5b522ed814f00a7d9bacfc73631fc" candidate_uuid: "0680b333153349cbbb6604efd4e60b96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0d2cc7de334901a8288347ef22e294"
I20260929 15:58:33.762274 13353 raft_consensus.cc:3037] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.762317 13634 raft_consensus.cc:3037] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:33.765893 13353 raft_consensus.cc:2454] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 1.
I20260929 15:58:33.765883 13634 raft_consensus.cc:2454] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0680b333153349cbbb6604efd4e60b96 in term 1.
I20260929 15:58:33.766532 13425 leader_election.cc:302] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0680b333153349cbbb6604efd4e60b96, 08df1fe400154ddf85da1c41bde60727, cf0d2cc7de334901a8288347ef22e294; no voters:
I20260929 15:58:33.767040 13739 raft_consensus.cc:2781] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:33.767596 13739 raft_consensus.cc:686] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 1 LEADER]: Becoming Leader. State: Replica: 0680b333153349cbbb6604efd4e60b96, State: Running, Role: LEADER
I20260929 15:58:33.768313 13739 consensus_queue.cc:237] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } }
I20260929 15:58:33.774344 13197 catalog_manager.cc:5495] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0680b333153349cbbb6604efd4e60b96 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "0680b333153349cbbb6604efd4e60b96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0680b333153349cbbb6604efd4e60b96" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:33.850414 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.838840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.5.117.2:39747)
0929 15:58:33.844259 (+ 5419us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.844265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.844288 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:58:33.844372 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.844376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.844458 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.844576 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:33.845229 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.846009 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.848205 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.848710 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.848818 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.849488 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:33.849632 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:33.849722 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.849915 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.849999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5281,"thread_start_us":5148,"threads_started":1}
I20260929 15:58:33.862383 13634 raft_consensus.cc:1261] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Refusing update from remote peer 0680b333153349cbbb6604efd4e60b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:33.862682 13353 raft_consensus.cc:1261] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Refusing update from remote peer 0680b333153349cbbb6604efd4e60b96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:33.863425 13817 consensus_queue.cc:1035] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08df1fe400154ddf85da1c41bde60727" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:33.863896 13739 consensus_queue.cc:1035] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf0d2cc7de334901a8288347ef22e294" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:33.932821 13196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:33.933163 13196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:33.936493 13196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a666f49218f74501819a35383cdb5482: Sending DeleteTablet for 3 replicas of tablet 79f5b522ed814f00a7d9bacfc73631fc
I20260929 15:58:33.937399 13334 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52926
I20260929 15:58:33.937633 13474 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260929 15:58:33.937911 13613 tablet_service.cc:1514] Processing DeleteTablet for tablet 79f5b522ed814f00a7d9bacfc73631fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:33 UTC) from {username='oryx', principal='oryx/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:32976
I20260929 15:58:33.938416 13780 tablet_replica.cc:330] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: stopping tablet replica
I20260929 15:58:33.938441 13827 tablet_replica.cc:330] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: stopping tablet replica
I20260929 15:58:33.938583 13780 raft_consensus.cc:2229] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:33.938632 13827 raft_consensus.cc:2229] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:33.938860 13827 raft_consensus.cc:2258] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.938894 13780 raft_consensus.cc:2258] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.940158 13780 ts_tablet_manager.cc:1905] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.940905 13827 ts_tablet_manager.cc:1905] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.941744 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13268
I20260929 15:58:33.949046 13828 tablet_replica.cc:330] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: stopping tablet replica
I20260929 15:58:33.949399 13828 raft_consensus.cc:2229] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:33.949651 13827 ts_tablet_manager.cc:1918] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:33.949720 13828 raft_consensus.cc:2258] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:33.949859 13827 log.cc:1196] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/79f5b522ed814f00a7d9bacfc73631fc
I20260929 15:58:33.949923 13780 ts_tablet_manager.cc:1918] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:33.950156 13780 log.cc:1196] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/79f5b522ed814f00a7d9bacfc73631fc
I20260929 15:58:33.950361 13827 ts_tablet_manager.cc:1939] T 79f5b522ed814f00a7d9bacfc73631fc P 08df1fe400154ddf85da1c41bde60727: Deleting consensus metadata
I20260929 15:58:33.950698 13780 ts_tablet_manager.cc:1939] T 79f5b522ed814f00a7d9bacfc73631fc P 0680b333153349cbbb6604efd4e60b96: Deleting consensus metadata
I20260929 15:58:33.952003 13182 catalog_manager.cc:4841] TS 08df1fe400154ddf85da1c41bde60727 (127.5.117.1:36873): tablet 79f5b522ed814f00a7d9bacfc73631fc (table test-table [id=9166227f64534bb7bb6750a5ed3860d0]) successfully deleted
I20260929 15:58:33.952000 13181 catalog_manager.cc:4841] TS 0680b333153349cbbb6604efd4e60b96 (127.5.117.2:39747): tablet 79f5b522ed814f00a7d9bacfc73631fc (table test-table [id=9166227f64534bb7bb6750a5ed3860d0]) successfully deleted
I20260929 15:58:33.952755 13828 ts_tablet_manager.cc:1905] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:33.957959 13828 ts_tablet_manager.cc:1918] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:33.958086 13828 log.cc:1196] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/79f5b522ed814f00a7d9bacfc73631fc
I20260929 15:58:33.958518 13828 ts_tablet_manager.cc:1939] T 79f5b522ed814f00a7d9bacfc73631fc P cf0d2cc7de334901a8288347ef22e294: Deleting consensus metadata
I20260929 15:58:33.959717 13181 catalog_manager.cc:4841] TS cf0d2cc7de334901a8288347ef22e294 (127.5.117.3:40651): tablet 79f5b522ed814f00a7d9bacfc73631fc (table test-table [id=9166227f64534bb7bb6750a5ed3860d0]) successfully deleted
Sep 29 15:58:33 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.1@KRBTEST.COM
I20260929 15:58:33.969134 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.943355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.5.117.1:36873)
0929 15:58:33.948595 (+ 5240us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.948603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.953163 (+ 4560us) server_negotiation.cc:407] Connection header received
0929 15:58:33.953400 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.953406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.953503 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.953649 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.954984 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.955834 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.957198 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.957624 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.957771 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.962524 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:33.962547 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:33.962572 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:33.962616 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:33.965861 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:33.966677 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:33.966686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:33.966688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:33.966830 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:33.967238 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:33.967243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:33.967245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:33.967703 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:33.968106 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:33.968439 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 15:58:33.968708 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5067,"thread_start_us":4934,"threads_started":1}
I20260929 15:58:33.969906 13399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:34.147665 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13268
I20260929 15:58:34.183003 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13408
Sep 29 15:58:34 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.2@KRBTEST.COM
I20260929 15:58:34.201088 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:34.184161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.5.117.2:39747)
0929 15:58:34.184403 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:34.184408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:34.184586 (+ 178us) server_negotiation.cc:407] Connection header received
0929 15:58:34.184761 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:34.184766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:34.184846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:34.184953 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:34.186439 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.187189 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.188688 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.189145 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.189250 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.194167 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:34.194179 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:34.194197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:34.194257 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:34.197600 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:34.198557 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:34.198564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:34.198566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:34.198717 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:34.199164 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:34.199168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:34.199170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:34.199541 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:34.199812 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:34.200587 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 15:58:34.200796 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:58:34.201819 13539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:34.351614 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13408
I20260929 15:58:34.379505 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13548
Sep 29 15:58:34 dist-test-slave-b69v krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.117.3@KRBTEST.COM
I20260929 15:58:34.397359 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:34.380457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.5.117.3:40651)
0929 15:58:34.381069 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:34.381074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:34.381092 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:34.381162 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:34.381167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:34.381258 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:34.381406 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:34.382933 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.383655 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.385163 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.385611 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.385762 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.391248 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:34.391271 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:34.391294 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:34.391350 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:34.394571 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:34.395317 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:34.395325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:34.395328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:34.395473 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:34.395811 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:34.395814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:34.395816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:34.396307 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:34.396555 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:34.396813 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 15:58:34.397016 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":326,"threads_started":1}
I20260929 15:58:34.398186 13679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:34.540247 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13548
I20260929 15:58:34.565626 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13165
I20260929 15:58:34.567198 13237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:34.702471 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13165
2026-09-29T15:58:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6871 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:34 dist-test-slave-b69v krb5kdc[13838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:34 dist-test-slave-b69v krb5kdc[13838](info): set up 2 sockets
Sep 29 15:58:34 dist-test-slave-b69v krb5kdc[13838](info): commencing operation
krb5kdc: starting...
W20260929 15:58:35.825518 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:36 dist-test-slave-b69v krb5kdc[13838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:58:36.111470 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45023
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:36.499856 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:36.500341 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:36.500527 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:36.508867 13851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:36.509022 13851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:36.509135 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:36.509243 13851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:36.509336 13851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:36.520906 13851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:36.522693 13851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:36.524407 13851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:36.538692 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:36.538817 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:36.538813 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:36.539805 13851 server_base.cc:1034] running on GCE node
I20260929 15:58:36.540829 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:36.542476 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:36.543707 13851 hybrid_clock.cc:648] HybridClock initialized: now 1790697516543644 us; error 53 us; skew 500 ppm
I20260929 15:58:36.544198 13851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T15:58:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1917 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:36 dist-test-slave-b69v krb5kdc[13865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:36 dist-test-slave-b69v krb5kdc[13865](info): set up 2 sockets
Sep 29 15:58:36 dist-test-slave-b69v krb5kdc[13865](info): commencing operation
krb5kdc: starting...
W20260929 15:58:37.745270 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:58:37 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:58:38.009606 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41347
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:38.382195 13878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:38.382521 13878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:38.382622 13878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:38.389886 13878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:38.390007 13878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:38.390058 13878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:38.390112 13878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:38.390161 13878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:38.400318 13878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:38.401818 13878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:38.403275 13878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:38.416626 13883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:38.416548 13886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:38.416519 13884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:38.417433 13878 server_base.cc:1034] running on GCE node
I20260929 15:58:38.418304 13878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:38.419808 13878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:38.421041 13878 hybrid_clock.cc:648] HybridClock initialized: now 1790697518420980 us; error 74 us; skew 500 ppm
I20260929 15:58:38.421415 13878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:38 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697518, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:38.430482 13878 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:58:38.432229 13878 webserver.cc:460] Webserver started at http://127.5.117.62:45549/ using document root <none> and password file <none>
I20260929 15:58:38.433044 13878 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:38.433195 13878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:38.433547 13878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:38.436818 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "67070e86ab43422c9eacd1ea2fb7ce9a"
format_stamp: "Formatted at 2026-09-29 15:58:38 on dist-test-slave-b69v"
server_key: "8a3c80fb5b102c84f601fe74e2a217e0"
server_key_iv: "25123fad22ab11da9264bef38ac6a213"
server_key_version: "encryptionkey@0"
I20260929 15:58:38.437618 13878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "67070e86ab43422c9eacd1ea2fb7ce9a"
format_stamp: "Formatted at 2026-09-29 15:58:38 on dist-test-slave-b69v"
server_key: "8a3c80fb5b102c84f601fe74e2a217e0"
server_key_iv: "25123fad22ab11da9264bef38ac6a213"
server_key_version: "encryptionkey@0"
I20260929 15:58:38.442895 13878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:58:38.446334 13893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:38.447681 13878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:38.447883 13878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "67070e86ab43422c9eacd1ea2fb7ce9a"
format_stamp: "Formatted at 2026-09-29 15:58:38 on dist-test-slave-b69v"
server_key: "8a3c80fb5b102c84f601fe74e2a217e0"
server_key_iv: "25123fad22ab11da9264bef38ac6a213"
server_key_version: "encryptionkey@0"
I20260929 15:58:38.448134 13878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:38.472765 13878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:38.477640 13878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:38.477982 13878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:38.500850 13878 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:41347
I20260929 15:58:38.500892 13944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:41347 every 8 connection(s)
I20260929 15:58:38.502398 13878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:38.503897 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13878
I20260929 15:58:38.504204 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:38.505177 5588 external_mini_cluster.cc:1428] Setting key a016aad1713a06aedc2bd45ec8883dca
I20260929 15:58:38.508149 13945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:58:38 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:38.523288 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Bootstrap starting.
I20260929 15:58:38.527100 13945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:38.528386 13945 log.cc:824] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:38.531543 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:38.507624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.5.117.62:41347)
0929 15:58:38.508539 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:38.508551 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:38.508605 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:58:38.510874 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:38.510915 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:38.511527 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:38.511984 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:38.514038 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.515511 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.517244 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.517913 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.518175 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.523369 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:38.523378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:38.523395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:38.523461 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:38.526825 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:38.527663 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:38.527671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:38.527678 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:38.527856 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:38.528338 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:38.528342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:38.528345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:38.529175 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:38.529558 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:38.529956 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 15:58:38.530463 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":452,"threads_started":1}
I20260929 15:58:38.532115 13945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: No bootstrap required, opened a new log
I20260929 15:58:38.540015 13945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:38.540521 13945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:38.540639 13945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67070e86ab43422c9eacd1ea2fb7ce9a, State: Initialized, Role: FOLLOWER
I20260929 15:58:38.541234 13945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:38.541497 13945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:38.541652 13945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:38.541838 13945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:38.544176 13945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:38.544729 13945 leader_election.cc:302] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67070e86ab43422c9eacd1ea2fb7ce9a; no voters:
I20260929 15:58:38.545498 13945 leader_election.cc:288] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:38.545713 13950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:38.546672 13950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 LEADER]: Becoming Leader. State: Replica: 67070e86ab43422c9eacd1ea2fb7ce9a, State: Running, Role: LEADER
I20260929 15:58:38.547286 13950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:38.547865 13945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:38.550544 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67070e86ab43422c9eacd1ea2fb7ce9a. Latest consensus state: current_term: 1 leader_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } } }
I20260929 15:58:38.550946 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:38.550628 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } } }
I20260929 15:58:38.551074 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:38.552763 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:38.557987 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:38.566819 13957 catalog_manager.cc:1347] Generated new cluster ID: 8af0071611b946deb699850873da81b3
I20260929 15:58:38.566938 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:38.587731 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:38.589890 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:38.598590 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Generated new TSK 0
I20260929 15:58:38.599608 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:58:38.660691 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:39.023958 13971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:39.024323 13971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:39.024426 13971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:39.031678 13971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:39.031774 13971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:39.031883 13971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:39.041599 13971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:39.043000 13971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:39.044646 13971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:39.059444 13976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:39.059419 13977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:39.059517 13979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:39.060439 13971 server_base.cc:1034] running on GCE node
I20260929 15:58:39.061034 13971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:39.062070 13971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:39.063323 13971 hybrid_clock.cc:648] HybridClock initialized: now 1790697519063286 us; error 60 us; skew 500 ppm
I20260929 15:58:39.063673 13971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:39 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:39.067278 13971 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:58:39.069072 13971 webserver.cc:460] Webserver started at http://127.5.117.1:34249/ using document root <none> and password file <none>
I20260929 15:58:39.069795 13971 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:39.069950 13971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:39.070309 13971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:39.073441 13971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "99cd63ad1f854170a022e23aa6bbed8a"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "e655e20452bf13c37fbb88fcbe643b9a"
server_key_iv: "b3c728613623b06595bd7bf507d2d0e5"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.074247 13971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "99cd63ad1f854170a022e23aa6bbed8a"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "e655e20452bf13c37fbb88fcbe643b9a"
server_key_iv: "b3c728613623b06595bd7bf507d2d0e5"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.079383 13971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 15:58:39.084209 13986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:39.085902 13971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:39.086108 13971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "99cd63ad1f854170a022e23aa6bbed8a"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "e655e20452bf13c37fbb88fcbe643b9a"
server_key_iv: "b3c728613623b06595bd7bf507d2d0e5"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.086328 13971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:39.127653 13971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:39.132562 13971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:39.132928 13971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:39.134114 13971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:39.135646 13971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:39.135758 13971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:39.135919 13971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:39.135998 13971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:39.174503 13971 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:42649
I20260929 15:58:39.174556 14098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:42649 every 8 connection(s)
I20260929 15:58:39.176075 13971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:39.184394 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 13971
I20260929 15:58:39.184662 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:39.185482 5588 external_mini_cluster.cc:1428] Setting key cc7fc82e789539e95591a2d6944e11b0
Sep 29 15:58:39 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:39.199028 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.179029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:52101 (local address 127.5.117.62:41347)
0929 15:58:39.179566 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:39.179573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:39.180650 (+ 1077us) server_negotiation.cc:407] Connection header received
0929 15:58:39.182502 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:39.182507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:39.182587 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:39.182687 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:39.184351 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.185526 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.187262 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.187739 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.187864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.191729 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:39.191737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:39.191744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:39.191784 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:39.195611 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:39.196420 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:39.196427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:39.196433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:39.196592 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:39.196997 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:39.197004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:39.197009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:39.197392 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:39.197549 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:39.198251 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 15:58:39.198513 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260929 15:58:39.199771 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.179641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:52101)
0929 15:58:39.180474 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:39.180527 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:39.182250 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:39.182890 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:39.182899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:39.183487 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:39.184177 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.184197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.185701 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.185705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.187122 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.187135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.187945 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.187950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.188211 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.188934 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:39.188956 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:39.191513 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:39.195747 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:39.195753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:39.195765 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:39.196294 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:39.196702 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:39.196705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:39.196707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:39.196881 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:39.197517 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:39.197523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:39.197846 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 15:58:39.198080 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 15:58:39.198495 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":372,"threads_started":1}
I20260929 15:58:39.201536 14099 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41347
I20260929 15:58:39.201876 14099 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:39.202665 14099 heartbeater.cc:511] Master 127.5.117.62:41347 requested a full tablet report, sending...
I20260929 15:58:39.204962 13910 ts_manager.cc:194] Registered new tserver with Master: 99cd63ad1f854170a022e23aa6bbed8a (127.5.117.1:42649)
I20260929 15:58:39.206493 13910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:52101
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:58:39.241279 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:39.614264 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:39.614646 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:39.614823 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:39.622357 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:39.622517 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:39.622697 14105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:39.632754 14105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:39.634346 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:39.635936 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:39.651343 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:39.651366 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:39.651314 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:39.652333 14105 server_base.cc:1034] running on GCE node
I20260929 15:58:39.653154 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:39.654408 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:39.655670 14105 hybrid_clock.cc:648] HybridClock initialized: now 1790697519655620 us; error 36 us; skew 500 ppm
I20260929 15:58:39.656001 14105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:39 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:39.659897 14105 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:58:39.661944 14105 webserver.cc:460] Webserver started at http://127.5.117.2:34245/ using document root <none> and password file <none>
I20260929 15:58:39.662768 14105 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:39.662932 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:39.663290 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:39.666529 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "5e737d3b061f403ba7ae34b5fb0bd476"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "7b9b22f17d4cacc034d41ec5731c5833"
server_key_iv: "39f78606741a1eb5c41c7d7736975ffb"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.667368 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "5e737d3b061f403ba7ae34b5fb0bd476"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "7b9b22f17d4cacc034d41ec5731c5833"
server_key_iv: "39f78606741a1eb5c41c7d7736975ffb"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.674444 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:58:39.678915 14120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:39.680769 14105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:39.680998 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "5e737d3b061f403ba7ae34b5fb0bd476"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "7b9b22f17d4cacc034d41ec5731c5833"
server_key_iv: "39f78606741a1eb5c41c7d7736975ffb"
server_key_version: "encryptionkey@0"
I20260929 15:58:39.681262 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:39.707294 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:39.712710 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:39.713088 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:39.714257 14105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:39.715775 14105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:39.715893 14105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:39.716080 14105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:39.716166 14105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 15:58:39.756934 14105 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:41929
I20260929 15:58:39.756992 14232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:41929 every 8 connection(s)
I20260929 15:58:39.758616 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:39.766088 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14105
I20260929 15:58:39.766372 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:39.767010 5588 external_mini_cluster.cc:1428] Setting key 51b108db576686ea1efe34ef59367219
Sep 29 15:58:39 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:39.786907 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.761921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:34699 (local address 127.5.117.62:41347)
0929 15:58:39.762490 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:39.762498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:39.763661 (+ 1163us) server_negotiation.cc:407] Connection header received
0929 15:58:39.765913 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:39.765918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:39.766024 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:39.766157 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:39.768611 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.769832 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.773013 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.773533 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.773656 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.778930 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:39.778936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:39.778940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:39.778979 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:39.782877 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:39.783840 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:39.783844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:39.783847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:39.784158 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:39.784717 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:39.784721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:39.784723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:39.785074 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:39.785200 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:39.786292 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0929 15:58:39.786541 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
I20260929 15:58:39.788677 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.762577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:34699)
0929 15:58:39.763451 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:39.763504 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:39.765483 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:39.766496 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:39.766511 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:39.767392 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:39.768376 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.768412 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.770217 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.770237 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.772798 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.772824 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.773823 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.773833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.774227 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.775133 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:39.775171 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:39.778594 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:39.783050 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:39.783061 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:39.783083 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:39.783650 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:39.784298 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:39.784307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:39.784313 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:39.784581 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:39.785209 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:39.785220 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:39.785776 (+ 556us) client_negotiation.cc:769] Sending connection context
0929 15:58:39.786167 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 15:58:39.786876 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":340,"threads_started":1}
I20260929 15:58:39.790632 14233 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41347
I20260929 15:58:39.791046 14233 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:39.791936 14233 heartbeater.cc:511] Master 127.5.117.62:41347 requested a full tablet report, sending...
I20260929 15:58:39.793735 13910 ts_manager.cc:194] Registered new tserver with Master: 5e737d3b061f403ba7ae34b5fb0bd476 (127.5.117.2:41929)
I20260929 15:58:39.794818 13910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:34699
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:58:39.836694 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:40.209434 14099 heartbeater.cc:503] Master 127.5.117.62:41347 was elected leader, sending a full tablet report...
W20260929 15:58:40.212088 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:40.212390 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:40.212568 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:40.219558 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:40.219688 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:40.219898 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:40.229210 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:40.230693 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:40.232230 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:40.246361 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:40.246390 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:40.246644 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:40.247437 14239 server_base.cc:1034] running on GCE node
I20260929 15:58:40.248024 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:40.249423 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:40.250672 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790697520250601 us; error 48 us; skew 500 ppm
I20260929 15:58:40.251111 14239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:40 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:40.255086 14239 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:58:40.256580 14239 webserver.cc:460] Webserver started at http://127.5.117.3:32829/ using document root <none> and password file <none>
I20260929 15:58:40.257285 14239 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:40.257391 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:40.257679 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:40.260854 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "28d42a2ba4674b16b054e94a9adb5b8c"
format_stamp: "Formatted at 2026-09-29 15:58:40 on dist-test-slave-b69v"
server_key: "fd3e7ac57395a41eaf8127414cfb9832"
server_key_iv: "61c3994a404bd0a57856c7bf1a9fb23b"
server_key_version: "encryptionkey@0"
I20260929 15:58:40.261620 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "28d42a2ba4674b16b054e94a9adb5b8c"
format_stamp: "Formatted at 2026-09-29 15:58:40 on dist-test-slave-b69v"
server_key: "fd3e7ac57395a41eaf8127414cfb9832"
server_key_iv: "61c3994a404bd0a57856c7bf1a9fb23b"
server_key_version: "encryptionkey@0"
I20260929 15:58:40.266752 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:40.270131 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:40.271510 14239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 15:58:40.271680 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "28d42a2ba4674b16b054e94a9adb5b8c"
format_stamp: "Formatted at 2026-09-29 15:58:40 on dist-test-slave-b69v"
server_key: "fd3e7ac57395a41eaf8127414cfb9832"
server_key_iv: "61c3994a404bd0a57856c7bf1a9fb23b"
server_key_version: "encryptionkey@0"
I20260929 15:58:40.271960 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:40.298017 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:40.302047 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:40.302295 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:40.303268 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:40.304693 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:40.304769 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:40.304857 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:40.304903 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:40.340502 14239 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:45549
I20260929 15:58:40.340556 14366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:45549 every 8 connection(s)
I20260929 15:58:40.341995 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:40.343921 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14239
I20260929 15:58:40.344189 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:40.344725 5588 external_mini_cluster.cc:1428] Setting key d71450ef59bf8e3485ab0d6b66d1b218
Sep 29 15:58:40 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:58:40.366621 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.345560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:60063 (local address 127.5.117.62:41347)
0929 15:58:40.346158 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.346168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.346988 (+ 820us) server_negotiation.cc:407] Connection header received
0929 15:58:40.349635 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.349639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.349720 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.349825 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.351452 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.352577 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.354667 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.355122 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.355248 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.359631 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:40.359636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:40.359639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:40.359673 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:40.363132 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.363943 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.363955 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.363959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.364174 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.364645 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.364651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.364654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.364979 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:40.365150 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:40.365955 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 15:58:40.366200 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260929 15:58:40.367879 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.345784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:60063)
0929 15:58:40.346789 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:40.346845 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:40.349276 (+ 2431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:40.350045 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:40.350053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:40.350624 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:40.351251 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.351269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.352773 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.352781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.354499 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.354514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.355538 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.355544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.355789 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.356743 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:40.356772 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:40.359361 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:40.363284 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:40.363294 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:40.363314 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:40.363797 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:40.364296 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:40.364301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:40.364305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:40.364523 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:40.365133 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:40.365142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:40.365526 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 15:58:40.365838 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 15:58:40.366453 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":377,"threads_started":1}
I20260929 15:58:40.369704 14367 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41347
I20260929 15:58:40.370092 14367 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:40.370890 14367 heartbeater.cc:511] Master 127.5.117.62:41347 requested a full tablet report, sending...
I20260929 15:58:40.372673 13910 ts_manager.cc:194] Registered new tserver with Master: 28d42a2ba4674b16b054e94a9adb5b8c (127.5.117.3:45549)
I20260929 15:58:40.373847 13910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:60063
I20260929 15:58:40.374775 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
Entry for principal oryx/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab.
WARNING: no policy specified for oryx/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
Entry for principal oryx/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab.
WARNING: no policy specified for oryx/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
Entry for principal oryx/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab.
I20260929 15:58:40.586978 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13971
Sep 29 15:58:40 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:58:40.612290 14378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.588625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.5.117.1:42649)
0929 15:58:40.589628 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.589642 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.589684 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:58:40.589787 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.589808 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.590227 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.590492 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.592412 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.594529 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.595949 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.597266 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.597437 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.602314 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:40.602338 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:40.602371 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:40.602444 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:40.608022 (+ 5578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.608896 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.608910 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.608916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.609140 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.609842 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.609849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.609855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.610449 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:40.610887 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:40.611134 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 15:58:40.611649 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":508,"threads_started":1}
I20260929 15:58:40.614090 14094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:40.707299 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13971
I20260929 15:58:40.729014 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14105
Sep 29 15:58:40 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:58:40.750217 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.730322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33556 (local address 127.5.117.2:41929)
0929 15:58:40.730923 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.730932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.730976 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:58:40.731077 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.731086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.731364 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.731528 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.733148 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.734324 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.735813 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.736721 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.736892 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.742266 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:40.742281 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:40.742303 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:40.742349 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:40.746235 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.747168 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.747179 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.747182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.747433 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.747942 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.747947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.747949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.748687 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:40.748988 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:40.749356 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:58:40.749725 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":316,"threads_started":1}
I20260929 15:58:40.751964 14228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:40.839619 14233 heartbeater.cc:503] Master 127.5.117.62:41347 was elected leader, sending a full tablet report...
I20260929 15:58:40.841347 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14105
W20260929 15:58:40.859858 13894 connection.cc:535] server connection from 127.5.117.2:34699 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:58:40.860186 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14239
Sep 29 15:58:40 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:58:40.877777 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.861281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.5.117.3:45549)
0929 15:58:40.861878 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.861898 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.861940 (+ 42us) server_negotiation.cc:407] Connection header received
0929 15:58:40.862013 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.862021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.862292 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.862485 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.863945 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.865087 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.866412 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.867109 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.867249 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.871756 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:40.871767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:40.871782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:40.871816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:40.875029 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.875679 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.875687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.875689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.875826 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:40.876213 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:40.876217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:40.876219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:40.876620 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:40.876988 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:40.877133 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 15:58:40.877403 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":293,"threads_started":1}
I20260929 15:58:40.879213 14362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:40.966471 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14239
I20260929 15:58:40.985042 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 13878
I20260929 15:58:40.986492 13940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:41.105775 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 13878
I20260929 15:58:41.131685 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41347
--webserver_interface=127.5.117.62
--webserver_port=45549
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:41.511170 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:41.511559 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:41.511718 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:41.519052 14385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:41.519212 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:41.519372 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:41.519487 14385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:41.519568 14385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:41.529786 14385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:41347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=45549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:41.531405 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:41.533068 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:41.545886 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:41.545913 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:41.545948 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:41.546801 14385 server_base.cc:1034] running on GCE node
I20260929 15:58:41.547629 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:41.549170 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:41.550401 14385 hybrid_clock.cc:648] HybridClock initialized: now 1790697521550350 us; error 54 us; skew 500 ppm
I20260929 15:58:41.550768 14385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:41 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:41.554203 14385 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:58:41.555712 14385 webserver.cc:460] Webserver started at http://127.5.117.62:45549/ using document root <none> and password file <none>
I20260929 15:58:41.556519 14385 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:41.556689 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:41.562391 14385 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:41.565428 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:41.566884 14385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:41.567080 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "67070e86ab43422c9eacd1ea2fb7ce9a"
format_stamp: "Formatted at 2026-09-29 15:58:38 on dist-test-slave-b69v"
server_key: "8a3c80fb5b102c84f601fe74e2a217e0"
server_key_iv: "25123fad22ab11da9264bef38ac6a213"
server_key_version: "encryptionkey@0"
I20260929 15:58:41.568256 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:41.621913 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:41.626685 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:41.627002 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:41.647809 14385 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:41347
I20260929 15:58:41.647830 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:41347 every 8 connection(s)
I20260929 15:58:41.649559 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:41.650535 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14385
I20260929 15:58:41.651546 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:42649
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:41347
--webserver_port=34249
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:41.657474 14452 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:41.660976 14452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Bootstrap starting.
I20260929 15:58:41.691895 14452 log.cc:824] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:41.705018 14452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:41.705801 14452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Bootstrap complete.
I20260929 15:58:41.720674 14452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:41.722189 14452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67070e86ab43422c9eacd1ea2fb7ce9a, State: Initialized, Role: FOLLOWER
I20260929 15:58:41.723049 14452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:41.723369 14452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:41.723522 14452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:41.723716 14452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:41.729733 14452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:41.730329 14452 leader_election.cc:302] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67070e86ab43422c9eacd1ea2fb7ce9a; no voters:
I20260929 15:58:41.731033 14452 leader_election.cc:288] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:58:41.731277 14456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:58:41.732394 14456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [term 2 LEADER]: Becoming Leader. State: Replica: 67070e86ab43422c9eacd1ea2fb7ce9a, State: Running, Role: LEADER
I20260929 15:58:41.733114 14456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } }
I20260929 15:58:41.733443 14452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:41.736181 14457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } } }
I20260929 15:58:41.736222 14458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67070e86ab43422c9eacd1ea2fb7ce9a. Latest consensus state: current_term: 2 leader_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67070e86ab43422c9eacd1ea2fb7ce9a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41347 } } }
I20260929 15:58:41.736588 14457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:41.736668 14458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:41.751364 14461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:41.758185 14461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:41.759702 14461 catalog_manager.cc:1259] Loaded cluster ID: 8af0071611b946deb699850873da81b3
I20260929 15:58:41.759874 14461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:41.764273 14461 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:41.765888 14461 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67070e86ab43422c9eacd1ea2fb7ce9a: Loaded TSK: 0
I20260929 15:58:41.767688 14461 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:58:42.069309 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:42.069763 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:42.069978 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:42.078297 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:42.078447 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:42.078644 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:42.089599 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.1.keytab
--principal=oryx/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:42649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=34249
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:42.091378 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:42.093072 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:42.108649 14478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:42.108670 14479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:42.109787 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:42.110040 14454 server_base.cc:1034] running on GCE node
I20260929 15:58:42.110893 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:42.112224 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:42.113555 14454 hybrid_clock.cc:648] HybridClock initialized: now 1790697522113365 us; error 177 us; skew 500 ppm
I20260929 15:58:42.113991 14454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697522, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:42.118331 14454 init.cc:377] Logged in from keytab as oryx/127.5.117.1@KRBTEST.COM (short username oryx)
I20260929 15:58:42.120020 14454 webserver.cc:460] Webserver started at http://127.5.117.1:34249/ using document root <none> and password file <none>
I20260929 15:58:42.120910 14454 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:42.121083 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:42.127863 14454 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:58:42.131035 14488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.132823 14454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:42.133050 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "99cd63ad1f854170a022e23aa6bbed8a"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "e655e20452bf13c37fbb88fcbe643b9a"
server_key_iv: "b3c728613623b06595bd7bf507d2d0e5"
server_key_version: "encryptionkey@0"
I20260929 15:58:42.134182 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:42.173636 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:42.178709 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:42.179054 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:42.180291 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:42.181825 14454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:42.181985 14454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.182125 14454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:42.182224 14454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.220665 14454 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:42649
I20260929 15:58:42.220731 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:42649 every 8 connection(s)
I20260929 15:58:42.222405 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:42.226543 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14454
I20260929 15:58:42.227427 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:41929
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:41347
--webserver_port=34245
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.246291 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.226320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:41903)
0929 15:58:42.227723 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.227772 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.229664 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.231288 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.231296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.231858 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.232534 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.232552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.235485 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.235490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.236827 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.236840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.238432 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.238441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.238766 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.239672 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.239711 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.244877 (+ 5166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1109,"thread_start_us":844,"threads_started":1}
W20260929 15:58:42.246306 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.226301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:41903 (local address 127.5.117.62:41347)
0929 15:58:42.227216 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.227233 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.227932 (+ 699us) server_negotiation.cc:407] Connection header received
0929 15:58:42.230298 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.230319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.230768 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.231156 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.232760 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.235318 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.236987 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.238112 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.238307 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.245236 (+ 6929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:41903: BlockingRecv error: recv got EOF from 127.5.117.1:41903 (error 108)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":339,"threads_started":1}
W20260929 15:58:42.247242 14601 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:41347 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.117.62:41347: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.259322 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.247876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:55031)
0929 15:58:42.248123 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.248146 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.248254 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.248570 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.248578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.249063 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.249307 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.249321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.250369 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.250377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.252133 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.252150 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.252912 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.252920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.253152 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.253790 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.253821 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.258994 (+ 5173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 15:58:42.259864 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.248126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:55031 (local address 127.5.117.62:41347)
0929 15:58:42.248295 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.248299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.248315 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:58:42.248356 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.248359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.248419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.248508 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.249487 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.250216 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.252280 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.252698 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.252858 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.259359 (+ 6501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:55031: BlockingRecv error: recv got EOF from 127.5.117.1:55031 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.279234 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.261358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:41849)
0929 15:58:42.261577 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.261613 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.261715 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.268890 (+ 7175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.268897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.269683 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.269905 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.269917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.271530 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.271536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.273218 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.273233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.274397 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.274402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.274612 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.275420 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.275445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.278982 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 15:58:42.279896 14601 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:58:42.280452 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.268298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:41849 (local address 127.5.117.62:41347)
0929 15:58:42.268516 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.268522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.268544 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:42.268606 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.268611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.268691 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.268804 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.270076 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.271366 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.273396 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.274049 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.274189 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.280261 (+ 6072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:41849: BlockingRecv error: recv got EOF from 127.5.117.1:41849 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 15:58:42.625226 14605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:42.625511 14605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:42.625661 14605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:42.632932 14605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:42.633054 14605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:42.633248 14605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:58:42.642750 14605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.2.keytab
--principal=oryx/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:41929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=34245
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:42.644032 14605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:42.645431 14605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:42.660677 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:42.660662 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:42.661129 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:42.661562 14605 server_base.cc:1034] running on GCE node
I20260929 15:58:42.662220 14605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:42.663259 14605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:42.664638 14605 hybrid_clock.cc:648] HybridClock initialized: now 1790697522664541 us; error 82 us; skew 500 ppm
I20260929 15:58:42.665266 14605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697522, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:42.669418 14605 init.cc:377] Logged in from keytab as oryx/127.5.117.2@KRBTEST.COM (short username oryx)
I20260929 15:58:42.671017 14605 webserver.cc:460] Webserver started at http://127.5.117.2:34245/ using document root <none> and password file <none>
I20260929 15:58:42.671846 14605 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:42.672091 14605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:42.677851 14605 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 15:58:42.680790 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.682303 14605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:42.682598 14605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "5e737d3b061f403ba7ae34b5fb0bd476"
format_stamp: "Formatted at 2026-09-29 15:58:39 on dist-test-slave-b69v"
server_key: "7b9b22f17d4cacc034d41ec5731c5833"
server_key_iv: "39f78606741a1eb5c41c7d7736975ffb"
server_key_version: "encryptionkey@0"
I20260929 15:58:42.683642 14605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:42.701987 14605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:42.707376 14605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:42.707690 14605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:42.709165 14605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:42.711074 14605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:42.711244 14605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.711388 14605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:42.711495 14605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:42.751379 14605 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:41929
I20260929 15:58:42.751518 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:41929 every 8 connection(s)
I20260929 15:58:42.753185 14605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:42.761032 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14605
I20260929 15:58:42.761729 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:45549
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:41347
--webserver_port=32829
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:42.771090 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.756253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:32795 (local address 127.5.117.62:41347)
0929 15:58:42.756459 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.756464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.757810 (+ 1346us) server_negotiation.cc:407] Connection header received
0929 15:58:42.759719 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.759723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.759802 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.760273 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.761888 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.762710 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.764406 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.764820 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.764983 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.770756 (+ 5773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:32795: BlockingRecv error: recv got EOF from 127.5.117.2:32795 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:42.772113 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.756822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:32795)
0929 15:58:42.757642 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.757687 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.759440 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.760460 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.760469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.761012 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.761694 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.761713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.762887 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.762897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.764239 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.764251 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.765059 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.765068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.765331 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.766082 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.766105 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.770522 (+ 4417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
W20260929 15:58:42.772882 14733 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:41347 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.117.62:41347: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.783041 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.773556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:58859)
0929 15:58:42.773821 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.773842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.773940 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.774248 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.774256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.774721 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.774933 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.774945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.776003 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.776011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.777690 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.777708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.778449 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.778458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.778685 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.779328 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.779355 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.782710 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 15:58:42.783244 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.773804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:58859 (local address 127.5.117.62:41347)
0929 15:58:42.773959 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.773963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.774004 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:58:42.774048 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.774051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.774118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.774195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.775093 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.775848 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.777836 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.778272 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.778388 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.782955 (+ 4567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:58859: BlockingRecv error: recv got EOF from 127.5.117.2:58859 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.792900 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.784019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:42321)
0929 15:58:42.784219 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.784240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.784332 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.784747 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.784751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.785076 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.785228 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.785235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.786852 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.786855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.787911 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.787923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.789112 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.789117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.789269 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.789904 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.789921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.792665 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 15:58:42.793267 14733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:58:42.793286 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.784167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:42321 (local address 127.5.117.62:41347)
0929 15:58:42.784350 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.784358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.784398 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:58:42.784465 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.784473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.784577 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.784690 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.785374 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.786659 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.788081 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.788965 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.789124 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.792985 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:42321: BlockingRecv error: recv got EOF from 127.5.117.2:42321 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 15:58:43.130862 14736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:43.131170 14736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:43.131268 14736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:43.138852 14736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:43.138970 14736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:43.139156 14736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:58:43.149153 14736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/krb5kdc/oryx_127.5.117.3.keytab
--principal=oryx/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:45549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=32829
--tserver_master_addrs=127.5.117.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:43.150609 14736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:43.152025 14736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:43.166560 14742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:43.166610 14741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:43.166760 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:43.167412 14736 server_base.cc:1034] running on GCE node
I20260929 15:58:43.168098 14736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:43.169279 14736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:43.170584 14736 hybrid_clock.cc:648] HybridClock initialized: now 1790697523170490 us; error 84 us; skew 500 ppm
I20260929 15:58:43.170955 14736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697523, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:43.174764 14736 init.cc:377] Logged in from keytab as oryx/127.5.117.3@KRBTEST.COM (short username oryx)
I20260929 15:58:43.176620 14736 webserver.cc:460] Webserver started at http://127.5.117.3:32829/ using document root <none> and password file <none>
I20260929 15:58:43.177426 14736 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:43.177605 14736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:43.183616 14736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:43.186800 14751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:43.188588 14736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:43.188786 14736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "28d42a2ba4674b16b054e94a9adb5b8c"
format_stamp: "Formatted at 2026-09-29 15:58:40 on dist-test-slave-b69v"
server_key: "fd3e7ac57395a41eaf8127414cfb9832"
server_key_iv: "61c3994a404bd0a57856c7bf1a9fb23b"
server_key_version: "encryptionkey@0"
I20260929 15:58:43.189934 14736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:43.216750 14736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:43.221743 14736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:43.222105 14736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:43.223318 14736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:43.224910 14736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:43.225004 14736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:43.225167 14736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:43.225242 14736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:43.266369 14736 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:45549
I20260929 15:58:43.266417 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:45549 every 8 connection(s)
I20260929 15:58:43.268652 14736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:43.273432 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14736
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:43.293898 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:43.272203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:54495 (local address 127.5.117.62:41347)
0929 15:58:43.272444 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.272449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.274002 (+ 1553us) server_negotiation.cc:407] Connection header received
0929 15:58:43.276423 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.276429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.276534 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.276646 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.278229 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.279229 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.282629 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.283847 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.283981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.293506 (+ 9525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:54495: BlockingRecv error: recv got EOF from 127.5.117.3:54495 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:58:43.295410 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.272838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:54495)
0929 15:58:43.273792 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:43.273841 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:43.276160 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:43.276853 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:43.276861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:43.277399 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:43.278055 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.278073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.279372 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.279379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.281989 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.282040 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.284659 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.284685 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.285317 (+ 632us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.286349 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.286405 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.293617 (+ 7212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":418,"threads_started":1}
W20260929 15:58:43.296386 14864 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:41347 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.117.62:41347: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:43.299091 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.280791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:45007)
0929 15:58:43.284569 (+ 3778us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:43.284598 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:43.284728 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:43.287750 (+ 3022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:43.287758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:43.289031 (+ 1273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:43.289305 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.289325 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.290890 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.290897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.292354 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.292372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.293688 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.293695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.293955 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.294828 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.294853 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.298725 (+ 3872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3621,"thread_start_us":3488,"threads_started":1}
W20260929 15:58:43.299425 14870 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:43.281053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45007 (local address 127.5.117.62:41347)
0929 15:58:43.287175 (+ 6122us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.287181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.287201 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:43.287261 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.287268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.287389 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.287527 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.289571 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.290747 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.292529 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.293222 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.293345 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.299094 (+ 5749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:45007: BlockingRecv error: recv got EOF from 127.5.117.1:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":5977,"thread_start_us":5801,"threads_started":1}
I20260929 15:58:43.300920 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.274708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.5.117.62:41347)
0929 15:58:43.275434 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.275439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.275456 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:58:43.275528 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.275532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.275602 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.275734 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.277668 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.278814 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.280857 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.282094 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.282273 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.290573 (+ 8300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:43.290585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:43.290608 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:43.290651 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:43.295141 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:43.296616 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:43.296626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:43.296636 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:43.296861 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:43.297449 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:43.297454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:43.297458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:43.298225 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:43.298476 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:43.300314 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0929 15:58:43.300652 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":412,"threads_started":1}
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:43.308384 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.297195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:50375)
0929 15:58:43.297486 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:43.297508 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:43.297635 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:43.298040 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:43.298046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:43.298514 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:43.298707 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.298719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.300094 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.300100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.301323 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.301335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.302290 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.302294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.302475 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.303452 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.303486 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.308022 (+ 4536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260929 15:58:43.308532 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:43.297264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:50375 (local address 127.5.117.62:41347)
0929 15:58:43.297502 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.297508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.297563 (+ 55us) server_negotiation.cc:407] Connection header received
0929 15:58:43.297757 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.297762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.297834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.297981 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.298878 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.299911 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.301473 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.302129 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.302258 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.308276 (+ 6018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:50375: BlockingRecv error: recv got EOF from 127.5.117.3:50375 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:43.318282 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.309465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:50889)
0929 15:58:43.309829 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:43.309858 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:43.309996 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:43.310346 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:43.310352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:43.310703 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:43.310866 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.310874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.311929 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.311936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.313006 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.313019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.313784 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.313789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.313977 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.314601 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.314619 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.317937 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 15:58:43.318392 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:43.309644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:50889 (local address 127.5.117.62:41347)
0929 15:58:43.309947 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.309952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.309971 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:58:43.310136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.310143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.310210 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.310318 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.311015 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.311786 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.313185 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.313633 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.313773 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.318130 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:50889: BlockingRecv error: recv got EOF from 127.5.117.3:50889 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
W20260929 15:58:43.318774 14864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:43.812486 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.794099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:36347)
0929 15:58:43.794757 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:43.794774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:43.794976 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:43.795423 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:43.795430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:43.796090 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:43.796401 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.796415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.797737 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.797742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.798775 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.798787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.799492 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.799497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.799625 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.808032 (+ 8407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.808114 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.812038 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":392,"threads_started":1}
W20260929 15:58:43.812621 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:43.794141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36347 (local address 127.5.117.62:41347)
0929 15:58:43.794428 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.794438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.794876 (+ 438us) server_negotiation.cc:407] Connection header received
0929 15:58:43.795141 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.795146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.795235 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.795328 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.796553 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.797585 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.798905 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.799327 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.799435 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.812355 (+ 12920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:36347: BlockingRecv error: recv got EOF from 127.5.117.2:36347 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 15:58:44.272195 14860 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.5.117.3:45549
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:44.310402 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.300208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:49083)
0929 15:58:44.301065 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:44.301091 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:44.301245 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:44.301813 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:44.301818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:44.302286 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:44.302465 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.302473 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.303739 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.303743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.304766 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.304778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.305721 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.305727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.305888 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.306641 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:44.306670 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:44.310052 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":580,"threads_started":1}
W20260929 15:58:44.310631 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:44.300317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:49083 (local address 127.5.117.62:41347)
0929 15:58:44.300564 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:44.300569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:44.301277 (+ 708us) server_negotiation.cc:407] Connection header received
0929 15:58:44.301394 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:44.301400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:44.301502 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:44.301702 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:44.302640 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.303570 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.304983 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.305424 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.305533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.310345 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:49083: BlockingRecv error: recv got EOF from 127.5.117.1:49083 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:44.329564 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.319636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:58375)
0929 15:58:44.320265 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:44.320285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:44.320437 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:44.320789 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:44.320793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:44.321366 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:44.321657 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.321669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.322929 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.322936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.324148 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.324166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.324910 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.324916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.325104 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.325798 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:44.325826 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:44.329310 (+ 3484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":323,"threads_started":1}
W20260929 15:58:44.329928 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:44.319890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:58375 (local address 127.5.117.62:41347)
0929 15:58:44.320222 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:44.320228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:44.320410 (+ 182us) server_negotiation.cc:407] Connection header received
0929 15:58:44.320558 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:44.320568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:44.320653 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:44.320748 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:44.321819 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.322767 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.324296 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.324718 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.324869 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.329512 (+ 4643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:58375: BlockingRecv error: recv got EOF from 127.5.117.3:58375 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:44 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:44.823006 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.813789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:45555)
0929 15:58:44.814442 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:44.814460 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:44.814603 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:44.814931 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:44.814934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:44.815394 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:44.815571 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.815581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.816663 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.816666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.817604 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.817615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.818306 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.818310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.818425 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.819154 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:44.819172 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:44.822708 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":366,"threads_started":1}
W20260929 15:58:44.823226 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:44.813889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:45555 (local address 127.5.117.62:41347)
0929 15:58:44.814133 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:44.814139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:44.814579 (+ 440us) server_negotiation.cc:407] Connection header received
0929 15:58:44.814715 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:44.814719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:44.814793 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:44.814885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:44.815698 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.816543 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.817723 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.818161 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.818260 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.822914 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:45555: BlockingRecv error: recv got EOF from 127.5.117.2:45555 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:45.320452 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.311654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:60069)
0929 15:58:45.312304 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:45.312334 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:45.312474 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:45.312793 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:45.312797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:45.313212 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:45.313404 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.313413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.314541 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.314546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.315524 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.315535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.316338 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.316342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.316490 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.317141 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:45.317159 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:45.320118 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":410,"threads_started":1}
W20260929 15:58:45.320927 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:45.311870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:60069 (local address 127.5.117.62:41347)
0929 15:58:45.312138 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.312146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.312466 (+ 320us) server_negotiation.cc:407] Connection header received
0929 15:58:45.312578 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.312583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.312657 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.312751 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.313589 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.314396 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.315697 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.316190 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.316318 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.320327 (+ 4009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:60069: BlockingRecv error: recv got EOF from 127.5.117.1:60069 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:45.338979 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.330688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:42837)
0929 15:58:45.331190 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:45.331207 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:45.331326 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:45.331627 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:45.331631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:45.332038 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:45.332256 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.332266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.333376 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.333383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.334593 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.334606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.335276 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.335280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.335386 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.335919 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:45.335936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:45.338688 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":300,"threads_started":1}
W20260929 15:58:45.339145 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:45.330731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:42837 (local address 127.5.117.62:41347)
0929 15:58:45.330946 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.330955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.331312 (+ 357us) server_negotiation.cc:407] Connection header received
0929 15:58:45.331442 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.331446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.331503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.331586 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.332375 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.333223 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.334711 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.335121 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.335213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.338856 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:42837: BlockingRecv error: recv got EOF from 127.5.117.3:42837 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:45 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:45.832826 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.824281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:48555)
0929 15:58:45.824878 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:45.824900 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:45.825091 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:45.825521 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:45.825525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:45.826043 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:45.826249 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.826258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.827291 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.827294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.828284 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.828295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.828919 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.828923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.829042 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.829696 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:45.829713 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:45.832530 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
W20260929 15:58:45.833036 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:45.824341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:48555 (local address 127.5.117.62:41347)
0929 15:58:45.824634 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.824642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.825031 (+ 389us) server_negotiation.cc:407] Connection header received
0929 15:58:45.825230 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.825237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.825324 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.825461 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.826380 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.827177 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.828413 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.828804 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.828893 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.832718 (+ 3825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:48555: BlockingRecv error: recv got EOF from 127.5.117.2:48555 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:46.332346 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:46.321811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:45811)
0929 15:58:46.322538 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:46.322563 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:46.322803 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:46.323449 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:46.323458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:46.324030 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:46.324320 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.324333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.325649 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.325654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.326729 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.326742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.327484 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.327498 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.327665 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.328538 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:46.328559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:46.331976 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
W20260929 15:58:46.332703 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:46.321980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45811 (local address 127.5.117.62:41347)
0929 15:58:46.322259 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:46.322269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:46.322692 (+ 423us) server_negotiation.cc:407] Connection header received
0929 15:58:46.323023 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:46.323030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:46.323193 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:46.323335 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:46.324531 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.325475 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.326877 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.327299 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.327428 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.332352 (+ 4924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:45811: BlockingRecv error: recv got EOF from 127.5.117.1:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:46.349511 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:46.340040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:43819)
0929 15:58:46.340611 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:46.340628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:46.340803 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:46.341172 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:46.341178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:46.341606 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:46.341797 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.341806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.342872 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.342879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.343809 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.343821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.344612 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.344617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.344802 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.345482 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:46.345501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:46.349119 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
W20260929 15:58:46.349916 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:46.340130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:43819 (local address 127.5.117.62:41347)
0929 15:58:46.340369 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:46.340378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:46.340717 (+ 339us) server_negotiation.cc:407] Connection header received
0929 15:58:46.340945 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:46.340952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:46.341035 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:46.341143 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:46.341944 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.342743 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.343948 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.344462 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.344608 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.349429 (+ 4821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:43819: BlockingRecv error: recv got EOF from 127.5.117.3:43819 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:46 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:46.843482 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:46.834238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:45371)
0929 15:58:46.834828 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:46.834847 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:46.834981 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:46.835305 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:46.835309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:46.835750 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:46.835929 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.835938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.837052 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.837057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.838054 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:46.838064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.838831 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.838836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:46.839002 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.839871 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:46.839898 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:46.843206 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":345,"threads_started":1}
W20260929 15:58:46.843819 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:46.834236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:45371 (local address 127.5.117.62:41347)
0929 15:58:46.834448 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:46.834454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:46.834932 (+ 478us) server_negotiation.cc:407] Connection header received
0929 15:58:46.835098 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:46.835101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:46.835171 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:46.835261 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:46.836095 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.836918 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.838218 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:46.838674 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:46.838778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:46.843481 (+ 4703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:45371: BlockingRecv error: recv got EOF from 127.5.117.2:45371 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:47.343906 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.333771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:50491)
0929 15:58:47.334399 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.334423 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.334578 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.335001 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.335005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.335554 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.335778 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.335787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.337193 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.337198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.338276 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.338287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.339340 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.339347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.339535 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.340308 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:47.340328 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.343542 (+ 3214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":307,"threads_started":1}
W20260929 15:58:47.344022 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:47.334169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:50491 (local address 127.5.117.62:41347)
0929 15:58:47.334411 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.334416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.334532 (+ 116us) server_negotiation.cc:407] Connection header received
0929 15:58:47.334703 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.334709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.334831 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.334949 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.336111 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.337000 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.338482 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.339091 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.339204 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.343785 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:50491: BlockingRecv error: recv got EOF from 127.5.117.1:50491 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:47.360327 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.350730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:34693)
0929 15:58:47.351326 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.351343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.351474 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.351813 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.351816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.352308 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.352550 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.352563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.353795 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.353804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.354942 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.354957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.355645 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.355649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.355783 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.356522 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:47.356541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.359920 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":329,"threads_started":1}
W20260929 15:58:47.360440 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:47.350812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:34693 (local address 127.5.117.62:41347)
0929 15:58:47.350986 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.350990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.351422 (+ 432us) server_negotiation.cc:407] Connection header received
0929 15:58:47.351608 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.351612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.351680 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.351767 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.352676 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.353643 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.355074 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.355513 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.355608 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.360171 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:34693: BlockingRecv error: recv got EOF from 127.5.117.3:34693 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:47 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:47.854717 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.844872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:49747)
0929 15:58:47.845735 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.845764 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.845936 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.846318 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.846324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.846850 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.847032 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.847042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.848211 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.848217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.849173 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:47.849185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.849972 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.849978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:47.850163 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.850853 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:47.850872 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.854309 (+ 3437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":569,"threads_started":1}
W20260929 15:58:47.854794 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:47.845226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:49747 (local address 127.5.117.62:41347)
0929 15:58:47.845705 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.845714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.845851 (+ 137us) server_negotiation.cc:407] Connection header received
0929 15:58:47.846072 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.846078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.846157 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.846272 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.847191 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.847984 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.849327 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:47.849812 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:47.849961 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:47.854545 (+ 4584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:49747: BlockingRecv error: recv got EOF from 127.5.117.2:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":274}
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:48.355494 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:48.345274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:41497)
0929 15:58:48.345994 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:48.346017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:48.346192 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:48.346649 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:48.346654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:48.347267 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:48.347502 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.347511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.348892 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.348896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.349842 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.349853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.350798 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.350803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.350922 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.351640 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:48.351661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:48.355233 (+ 3572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":394,"threads_started":1}
W20260929 15:58:48.355931 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:48.345380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:41497 (local address 127.5.117.62:41347)
0929 15:58:48.345651 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:48.345661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:48.346168 (+ 507us) server_negotiation.cc:407] Connection header received
0929 15:58:48.346349 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:48.346354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:48.346453 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:48.346579 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:48.347719 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.348745 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.350009 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.350652 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.350753 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.355597 (+ 4844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:41497: BlockingRecv error: recv got EOF from 127.5.117.1:41497 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:48.369716 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:48.361460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:50355)
0929 15:58:48.361969 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:48.361985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:48.362154 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:48.362483 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:48.362488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:48.362899 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:48.363069 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.363078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.364132 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.364136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.365054 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.365065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.365764 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.365769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.365932 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.366575 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:48.366592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:48.369458 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":270,"threads_started":1}
W20260929 15:58:48.370266 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:48.361508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:50355 (local address 127.5.117.62:41347)
0929 15:58:48.361713 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:48.361722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:48.362084 (+ 362us) server_negotiation.cc:407] Connection header received
0929 15:58:48.362277 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:48.362282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:48.362354 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:48.362456 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:48.363204 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.363952 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.365213 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.365618 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.365752 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.369979 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:50355: BlockingRecv error: recv got EOF from 127.5.117.3:50355 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:48 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:48.872486 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:48.855904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:33781)
0929 15:58:48.856521 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:48.856539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:48.856672 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:48.857071 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:48.857076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:48.857535 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:48.857718 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.857729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.858860 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.858865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.859830 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:48.859843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.860686 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.860692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:48.860834 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.868167 (+ 7333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:48.868186 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:48.872172 (+ 3986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":348,"threads_started":1}
W20260929 15:58:48.872912 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:48.855957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:33781 (local address 127.5.117.62:41347)
0929 15:58:48.856195 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:48.856200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:48.856632 (+ 432us) server_negotiation.cc:407] Connection header received
0929 15:58:48.856781 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:48.856784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:48.856865 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:48.856962 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:48.857867 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.858678 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.859990 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:48.860508 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:48.860624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:48.872580 (+ 11956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:33781: BlockingRecv error: recv got EOF from 127.5.117.2:33781 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:49.367182 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:49.356594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:49103)
0929 15:58:49.357338 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:49.357357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:49.357531 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:49.357896 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:49.357901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:49.358385 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:49.358679 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.358694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.360038 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.360043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.361154 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.361165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.362005 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.362009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.362147 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.362898 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:49.362916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:49.366874 (+ 3958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":396,"threads_started":1}
W20260929 15:58:49.367754 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:49.356714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:49103 (local address 127.5.117.62:41347)
0929 15:58:49.356998 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:49.357009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:49.357455 (+ 446us) server_negotiation.cc:407] Connection header received
0929 15:58:49.357672 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:49.357677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:49.357755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:49.357858 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:49.358987 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.359834 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.361328 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.361838 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.361946 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.367192 (+ 5246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:49103: BlockingRecv error: recv got EOF from 127.5.117.1:49103 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:49.379689 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:49.370778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:33519)
0929 15:58:49.371484 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:49.371501 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:49.371626 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:49.371925 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:49.371929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:49.372422 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:49.372607 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.372616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.373793 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.373797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.374781 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.374792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.375552 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.375557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.375689 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.376488 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:49.376506 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:49.379443 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":406,"threads_started":1}
W20260929 15:58:49.380020 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:49.370962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:33519 (local address 127.5.117.62:41347)
0929 15:58:49.371197 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:49.371202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:49.371624 (+ 422us) server_negotiation.cc:407] Connection header received
0929 15:58:49.371720 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:49.371723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:49.371802 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:49.371903 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:49.372758 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.373641 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.374919 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.375378 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.375474 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.379672 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:33519: BlockingRecv error: recv got EOF from 127.5.117.3:33519 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:49 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:49.883679 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:49.873715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:36035)
0929 15:58:49.874430 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:49.874455 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:49.874664 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:49.875068 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:49.875074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:49.875603 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:49.875795 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.875805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.876989 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.876999 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.877991 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:49.878003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.878792 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.878799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:49.878993 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.879741 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:49.879759 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:49.883275 (+ 3516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":336,"threads_started":1}
W20260929 15:58:49.883913 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:49.873818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36035 (local address 127.5.117.62:41347)
0929 15:58:49.874116 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:49.874127 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:49.874568 (+ 441us) server_negotiation.cc:407] Connection header received
0929 15:58:49.874835 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:49.874839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:49.874911 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:49.875025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:49.875952 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.876826 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.878159 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:49.878615 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:49.878761 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:49.883586 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:36035: BlockingRecv error: recv got EOF from 127.5.117.2:36035 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:50.378202 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.368518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:42333)
0929 15:58:50.369232 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:50.369248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:50.369435 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:50.369805 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:50.369810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:50.370319 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:50.370504 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.370514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.371701 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.371708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.372953 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.372975 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.373817 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.373823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.374015 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.374717 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:50.374735 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:50.377928 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":409,"threads_started":1}
W20260929 15:58:50.378440 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:50.368630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:42333 (local address 127.5.117.62:41347)
0929 15:58:50.369017 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:50.369026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:50.369352 (+ 326us) server_negotiation.cc:407] Connection header received
0929 15:58:50.369590 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:50.369594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:50.369664 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:50.369777 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:50.370670 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.371560 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.373109 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.373612 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.373754 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.378159 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:42333: BlockingRecv error: recv got EOF from 127.5.117.1:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":194}
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:50.390892 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.380718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:36475)
0929 15:58:50.381363 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:50.381380 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:50.381496 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:50.381954 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:50.381957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:50.382379 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:50.382561 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.382570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.383725 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.383731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.384950 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.384967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.385773 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.385780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.385942 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.386776 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:50.386800 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:50.390472 (+ 3672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":315,"threads_started":1}
W20260929 15:58:50.390990 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:50.380836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:36475 (local address 127.5.117.62:41347)
0929 15:58:50.381039 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:50.381048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:50.381460 (+ 412us) server_negotiation.cc:407] Connection header received
0929 15:58:50.381698 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:50.381703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:50.381790 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:50.381915 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:50.382706 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.383470 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.385116 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.385591 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.385736 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.390743 (+ 5007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:36475: BlockingRecv error: recv got EOF from 127.5.117.3:36475 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:50 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:50.894460 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.885025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:52681)
0929 15:58:50.885688 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:50.885710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:50.885845 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:50.886173 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:50.886177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:50.886637 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:50.886826 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.886835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.888286 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.888292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.889271 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.889283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.889985 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.889990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.890132 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.890800 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:50.890820 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:50.894182 (+ 3362us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":321,"threads_started":1}
W20260929 15:58:50.894784 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:50.885131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:52681 (local address 127.5.117.62:41347)
0929 15:58:50.885390 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:50.885398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:50.885793 (+ 395us) server_negotiation.cc:407] Connection header received
0929 15:58:50.885952 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:50.885956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:50.886040 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:50.886126 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:50.887010 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.888133 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.889395 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.889824 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.889922 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.894378 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:52681: BlockingRecv error: recv got EOF from 127.5.117.2:52681 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:51.388538 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.379283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:44305)
0929 15:58:51.379928 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.379945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.380152 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.380527 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.380532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.381012 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:51.381198 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.381207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.382269 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.382275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.383264 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.383274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.384017 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.384022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.384242 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.384888 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:51.384908 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:51.388257 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":370,"threads_started":1}
W20260929 15:58:51.388736 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:51.379511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:44305 (local address 127.5.117.62:41347)
0929 15:58:51.379772 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.379779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.380045 (+ 266us) server_negotiation.cc:407] Connection header received
0929 15:58:51.380287 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.380293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.380376 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.380491 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.381343 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.382115 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.383420 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.383858 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.384003 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.388461 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:44305: BlockingRecv error: recv got EOF from 127.5.117.1:44305 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:51.401343 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.391896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:59597)
0929 15:58:51.392517 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.392535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.392657 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.392969 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.392972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.393361 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:51.393519 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.393527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.394794 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.394799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.395915 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.395926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.396870 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.396875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.397007 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.397913 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:51.397940 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:51.401096 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":348,"threads_started":1}
W20260929 15:58:51.401592 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:51.392151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:59597 (local address 127.5.117.62:41347)
0929 15:58:51.392361 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.392367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.392621 (+ 254us) server_negotiation.cc:407] Connection header received
0929 15:58:51.392760 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.392763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.392839 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.392922 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.393651 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.394621 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.396036 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.396684 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.396780 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.401302 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:59597: BlockingRecv error: recv got EOF from 127.5.117.3:59597 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:51 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:51.904861 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.895449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:55709)
0929 15:58:51.896104 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.896123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.896258 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.896609 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.896613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.897089 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:51.897278 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.897289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.898645 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.898651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.899947 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.899962 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.900696 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.900700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.900823 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.901539 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:51.901558 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:51.904592 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":346,"threads_started":1}
W20260929 15:58:51.905025 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:51.895542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:55709 (local address 127.5.117.62:41347)
0929 15:58:51.895734 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.895740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.896210 (+ 470us) server_negotiation.cc:407] Connection header received
0929 15:58:51.896364 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.896368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.896462 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.896553 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.897411 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.898480 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.900122 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.900565 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.900662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.904788 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:55709: BlockingRecv error: recv got EOF from 127.5.117.2:55709 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:52.399559 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.389666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:45277)
0929 15:58:52.390352 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:52.390368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:52.390540 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:52.390902 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:52.390908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:52.391375 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:52.391546 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.391554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.392707 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.392713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.393739 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.393750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.394526 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.394532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.394712 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.395698 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:52.395727 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:52.399269 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":429,"threads_started":1}
W20260929 15:58:52.399854 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:52.389723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45277 (local address 127.5.117.62:41347)
0929 15:58:52.389943 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.389948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.390461 (+ 513us) server_negotiation.cc:407] Connection header received
0929 15:58:52.390675 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.390681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.390761 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.390869 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.391692 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.392564 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.393883 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.394361 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.394512 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.399501 (+ 4989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:45277: BlockingRecv error: recv got EOF from 127.5.117.1:45277 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:52.411607 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.402494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:35613)
0929 15:58:52.403323 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:52.403340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:52.403463 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:52.403892 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:52.403896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:52.404402 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:52.404586 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.404595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.405636 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.405641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.406604 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.406616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.407417 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.407423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.407622 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.408370 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:52.408401 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:52.411336 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":565,"threads_started":1}
W20260929 15:58:52.411757 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:52.402582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:35613 (local address 127.5.117.62:41347)
0929 15:58:52.402895 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.402904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.403445 (+ 541us) server_negotiation.cc:407] Connection header received
0929 15:58:52.403628 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.403632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.403709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.403857 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.404756 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.405496 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.406744 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.407164 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.407273 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.411546 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:35613: BlockingRecv error: recv got EOF from 127.5.117.3:35613 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:52 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:52.916435 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.905995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:33035)
0929 15:58:52.906620 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:52.906645 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:52.906781 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:52.907307 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:52.907311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:52.907767 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:52.907944 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.907954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.909874 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.909881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.911076 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.911094 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.911919 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.911925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.912173 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.912883 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:52.912902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:52.916131 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":368,"threads_started":1}
W20260929 15:58:52.916692 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:52.906053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:33035 (local address 127.5.117.62:41347)
0929 15:58:52.906263 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.906269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.906821 (+ 552us) server_negotiation.cc:407] Connection header received
0929 15:58:52.906923 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.906928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.907051 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.907251 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.908234 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.909445 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.911236 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.911701 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.911804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.916369 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:33035: BlockingRecv error: recv got EOF from 127.5.117.2:33035 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:53.410621 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:53.400613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:53863)
0929 15:58:53.401246 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:53.401263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:53.401397 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:53.401718 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:53.401723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:53.402195 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:53.402411 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.402420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.404104 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.404113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.405155 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.405168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.405913 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.405917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.406044 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.406813 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:53.406831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:53.410325 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":358,"threads_started":1}
W20260929 15:58:53.411119 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:53.400824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:53863 (local address 127.5.117.62:41347)
0929 15:58:53.401045 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:53.401050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:53.401377 (+ 327us) server_negotiation.cc:407] Connection header received
0929 15:58:53.401503 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:53.401507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:53.401588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:53.401668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:53.402733 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.403885 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.405289 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.405754 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.405853 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.410724 (+ 4871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:53863: BlockingRecv error: recv got EOF from 127.5.117.1:53863 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:53.422143 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:53.412804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:34065)
0929 15:58:53.413378 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:53.413396 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:53.413520 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:53.413831 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:53.413835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:53.414245 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:53.414403 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.414411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.415680 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.415687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.416919 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.416931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.417645 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.417651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.417825 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.418626 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:53.418654 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:53.421874 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":338,"threads_started":1}
W20260929 15:58:53.422674 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:53.412901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:34065 (local address 127.5.117.62:41347)
0929 15:58:53.413155 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:53.413165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:53.413487 (+ 322us) server_negotiation.cc:407] Connection header received
0929 15:58:53.413631 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:53.413634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:53.413701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:53.413790 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:53.414525 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.415530 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.417076 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.417491 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.417632 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.422381 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:34065: BlockingRecv error: recv got EOF from 127.5.117.3:34065 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:53 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:53.932981 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:53.917662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:49389)
0929 15:58:53.918235 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:53.918252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:53.918450 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:53.918842 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:53.918847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:53.919365 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:53.919574 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.919582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.920843 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.920850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.921806 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:53.921818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.922583 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.922588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:53.922743 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.929287 (+ 6544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:53.929318 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:53.932679 (+ 3361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
W20260929 15:58:53.933350 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:53.917738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:49389 (local address 127.5.117.62:41347)
0929 15:58:53.918013 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:53.918023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:53.918352 (+ 329us) server_negotiation.cc:407] Connection header received
0929 15:58:53.918590 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:53.918596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:53.918693 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:53.918806 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:53.919736 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.920700 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.921953 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:53.922378 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:53.922558 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:53.932926 (+ 10368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:49389: BlockingRecv error: recv got EOF from 127.5.117.2:49389 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:54.421967 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:54.411730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:41965)
0929 15:58:54.412574 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:54.412592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:54.412788 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:54.413197 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:54.413203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:54.413711 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:54.413897 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.413906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.415015 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.415020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.415989 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.415999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.416813 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.416817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.416945 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.417898 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:54.417931 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:54.421538 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":574,"threads_started":1}
W20260929 15:58:54.422066 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:54.411890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:41965 (local address 127.5.117.62:41347)
0929 15:58:54.412275 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:54.412287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:54.412694 (+ 407us) server_negotiation.cc:407] Connection header received
0929 15:58:54.412955 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:54.412965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:54.413040 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:54.413151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:54.414041 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.414874 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.416219 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.416670 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.416779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.421798 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:41965: BlockingRecv error: recv got EOF from 127.5.117.1:41965 (error 108)
Metrics: {"server-negotiator.queue_time_us":199}
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:54.433027 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:54.423195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:43257)
0929 15:58:54.423804 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:54.423821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:54.423939 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:54.424286 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:54.424298 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:54.424709 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:54.424908 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.424915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.426002 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.426007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.426962 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.426974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.427805 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.427812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.427992 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.428808 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:54.428826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:54.432574 (+ 3748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":345,"threads_started":1}
W20260929 15:58:54.433147 14603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:54.423246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:43257 (local address 127.5.117.62:41347)
0929 15:58:54.423409 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:54.423413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:54.423899 (+ 486us) server_negotiation.cc:407] Connection header received
0929 15:58:54.424039 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:54.424042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:54.424154 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:54.424238 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:54.425061 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.425861 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.427092 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.427556 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.427653 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.432862 (+ 5209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:43257: BlockingRecv error: recv got EOF from 127.5.117.3:43257 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:54 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:54.944208 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:54.934169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:47069)
0929 15:58:54.934952 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:54.934976 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:54.935111 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:54.935528 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:54.935532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:54.935993 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:54.936259 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.936268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.937661 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.937665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.938726 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:54.938738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.939491 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.939496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:54.939628 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.940411 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:54.940430 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:54.943738 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":421,"threads_started":1}
W20260929 15:58:54.944823 14905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:54.934389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:47069 (local address 127.5.117.62:41347)
0929 15:58:54.935059 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:54.935064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:54.935082 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:54.935223 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:54.935227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:54.935319 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:54.935422 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:54.936482 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.937476 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.938868 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:54.939327 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:54.939437 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:54.943979 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:47069: BlockingRecv error: recv got EOF from 127.5.117.2:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":454,"threads_started":1}
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:55.432580 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:55.423375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:36655)
0929 15:58:55.424032 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:55.424076 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:55.424238 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:55.424722 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:55.424727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:55.425264 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:55.425506 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.425519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.426818 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.426822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.427748 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.427758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.428461 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.428465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.428580 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.429227 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:55.429245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:55.432303 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
W20260929 15:58:55.432763 14905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:55.423551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:36655 (local address 127.5.117.62:41347)
0929 15:58:55.423797 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:55.423803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:55.424171 (+ 368us) server_negotiation.cc:407] Connection header received
0929 15:58:55.424429 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:55.424435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:55.424533 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:55.424639 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:55.425664 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.426695 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.427865 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.428307 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.428393 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.432538 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:36655: BlockingRecv error: recv got EOF from 127.5.117.1:36655 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:55.443177 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:55.434498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:47477)
0929 15:58:55.435065 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:55.435082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:55.435252 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:55.435636 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:55.435640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:55.436023 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:55.436256 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.436265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.437380 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.437386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.438562 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.438576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.439341 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.439346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.439472 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.440134 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:55.440153 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:55.442910 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":323,"threads_started":1}
W20260929 15:58:55.443397 14905 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:55.434522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:47477 (local address 127.5.117.62:41347)
0929 15:58:55.434707 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:55.434716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:55.435171 (+ 455us) server_negotiation.cc:407] Connection header received
0929 15:58:55.435427 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:55.435431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:55.435508 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:55.435593 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:55.436375 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.437213 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.438742 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.439217 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.439313 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.443112 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:47477: BlockingRecv error: recv got EOF from 127.5.117.3:47477 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:55 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:55.954998 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:55.945560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:36779)
0929 15:58:55.946254 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:55.946282 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:55.946436 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:55.946851 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:55.946856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:55.947353 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:55.947536 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.947544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.948756 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.948763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.949765 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:55.949776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.950567 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.950573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:55.950759 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.951459 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:55.951477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:55.954601 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":351,"threads_started":1}
W20260929 15:58:55.955142 14909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:55.945735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36779 (local address 127.5.117.62:41347)
0929 15:58:55.946492 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:55.946501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:55.946521 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:55.946593 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:55.946599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:55.946678 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:55.946810 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:55.947686 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.948613 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.949925 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:55.950368 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:55.950531 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:55.954853 (+ 4322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:36779: BlockingRecv error: recv got EOF from 127.5.117.2:36779 (error 108)
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":500,"threads_started":1}
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:56.443014 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:56.433683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:41483)
0929 15:58:56.434325 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:56.434347 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:56.434537 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:56.434928 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:56.434934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:56.435578 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:56.435875 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.435888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.437062 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.437065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.438087 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.438107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.438796 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.438800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.438915 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.439629 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:56.439646 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:56.442620 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":334,"threads_started":1}
W20260929 15:58:56.443100 14909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:56.433849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:41483 (local address 127.5.117.62:41347)
0929 15:58:56.434123 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:56.434132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:56.434477 (+ 345us) server_negotiation.cc:407] Connection header received
0929 15:58:56.434683 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:56.434688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:56.434787 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:56.434917 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:56.436036 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.436930 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.438220 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.438680 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.438772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.442840 (+ 4068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:41483: BlockingRecv error: recv got EOF from 127.5.117.1:41483 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:56.452687 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:56.444272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:59177)
0929 15:58:56.444851 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:56.444868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:56.444994 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:56.445275 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:56.445278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:56.445659 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:56.445819 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.445826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.446831 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.446836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.447776 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.447787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.448673 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.448679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.448827 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.449469 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:56.449487 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:56.452289 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":332,"threads_started":1}
W20260929 15:58:56.452769 14909 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:56.444327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:59177 (local address 127.5.117.62:41347)
0929 15:58:56.444573 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:56.444577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:56.444948 (+ 371us) server_negotiation.cc:407] Connection header received
0929 15:58:56.445085 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:56.445088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:56.445157 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:56.445234 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:56.445929 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.446710 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.447919 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.448492 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.448622 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.452478 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:59177: BlockingRecv error: recv got EOF from 127.5.117.3:59177 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:56 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:56.966091 14913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:56.956297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:33251 (local address 127.5.117.62:41347)
0929 15:58:56.957223 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:56.957231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:56.957252 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:58:56.957315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:56.957320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:56.957440 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:56.957580 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:56.958599 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.959391 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.960781 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.961233 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.961376 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.965824 (+ 4448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:33251: BlockingRecv error: recv got EOF from 127.5.117.2:33251 (error 108)
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":679,"threads_started":1}
I20260929 15:58:56.966316 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:56.956252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:33251)
0929 15:58:56.956866 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:56.956890 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:56.957053 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:56.957624 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:56.957629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:56.958159 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:56.958436 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.958448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.959552 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.959563 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.960635 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:56.960648 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:56.961389 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:56.961395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:56.961595 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:56.962382 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:56.962401 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:56.965592 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.1@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:57.454504 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:57.444726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.1:45351)
0929 15:58:57.445533 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:57.445552 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:57.445679 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:57.446070 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:57.446074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:57.446569 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:57.446803 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.446812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.447984 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.447988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.449010 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.449021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.449778 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.449782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.449917 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.450621 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:57.450640 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:57.454065 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":530,"threads_started":1}
W20260929 15:58:57.454562 14913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:57.444788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:45351 (local address 127.5.117.62:41347)
0929 15:58:57.445058 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:57.445068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:57.445725 (+ 657us) server_negotiation.cc:407] Connection header received
0929 15:58:57.445807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:57.445812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:57.445894 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:57.446025 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:57.446989 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.447812 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.449175 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.449635 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.449743 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.454327 (+ 4584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.1:45351: BlockingRecv error: recv got EOF from 127.5.117.1:45351 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.3@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:57.463877 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:57.453934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.3:39663)
0929 15:58:57.455379 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:57.455424 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:57.455586 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:57.456013 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:57.456019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:57.456557 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:57.456827 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.456839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.458119 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.458123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.459057 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.459068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.459805 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.459809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.459929 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.460645 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:57.460664 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:57.463576 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1303,"thread_start_us":1148,"threads_started":1}
W20260929 15:58:57.464625 14913 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:57.453979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:39663 (local address 127.5.117.62:41347)
0929 15:58:57.454817 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:57.454823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:57.455531 (+ 708us) server_negotiation.cc:407] Connection header received
0929 15:58:57.455714 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:57.455718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:57.455830 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:57.455945 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:57.457009 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.457985 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.459248 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.459676 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.459777 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.464346 (+ 4569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.3:39663: BlockingRecv error: recv got EOF from 127.5.117.3:39663 (error 108)
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":1168,"threads_started":1}
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:57 dist-test-slave-b69v krb5kdc[13865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.117.2@KRBTEST.COM for oryx/127.5.117.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:57.977909 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:57.967936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41347 (local address 127.5.117.2:58025)
0929 15:58:57.968675 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:57.968693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:57.968823 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:57.969205 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:57.969209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:57.969732 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:57.969959 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.969968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.971310 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.971314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.972409 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:57.972420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.973500 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.973506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:57.973728 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.974581 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:57.974600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:57.977609 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.117.62:41347: Server oryx/127.5.117.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":473,"threads_started":1}
W20260929 15:58:57.978132 14918 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:57.968027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:58025 (local address 127.5.117.62:41347)
0929 15:58:57.968612 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:57.968618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:57.968778 (+ 160us) server_negotiation.cc:407] Connection header received
0929 15:58:57.968935 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:57.968939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:57.969014 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:57.969111 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:57.970245 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.971132 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.972562 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:57.973325 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:57.973474 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:57.977842 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.117.2:58025: BlockingRecv error: recv got EOF from 127.5.117.2:58025 (error 108)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":292,"threads_started":1}
I20260929 15:58:58.280932 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14454
I20260929 15:58:58.295605 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:58.282284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.5.117.1:42649)
0929 15:58:58.283020 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:58.283029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:58.283060 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:58:58.283134 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:58.283142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:58.283429 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:58.283616 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:58.285442 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.286615 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.288180 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.288861 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.289041 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:58.291722 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:58.291730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:58.291746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:58.291783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:58.295052 (+ 3269us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:58.295350 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60542: Request ticket server kudu/127.5.117.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":397,"threads_started":1}
W20260929 15:58:58.295756 14920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60542: Request ticket server kudu/127.5.117.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:58.295797 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14454: Not authorized: Client connection negotiation failed: client connection to 127.5.117.1:42649: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:58.296018 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14454
I20260929 15:58:58.316205 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14605
I20260929 15:58:58.330333 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:58.317578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.5.117.2:41929)
0929 15:58:58.318239 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:58.318248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:58.318282 (+ 34us) server_negotiation.cc:407] Connection header received
0929 15:58:58.318363 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:58.318370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:58.318628 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:58.318812 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:58.320043 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.321280 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.322458 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.323056 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.323164 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:58.325972 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:58.325984 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:58.326001 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:58.326055 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:58.329439 (+ 3384us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:58.329860 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51414: Request ticket server kudu/127.5.117.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":312,"threads_started":1}
W20260929 15:58:58.330456 14921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51414: Request ticket server kudu/127.5.117.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:58.330559 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14605: Not authorized: Client connection negotiation failed: client connection to 127.5.117.2:41929: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:58.330762 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14605
I20260929 15:58:58.351431 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14736
I20260929 15:58:58.365221 14922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:58.352733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.5.117.3:45549)
0929 15:58:58.353468 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:58.353477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:58.353507 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:58:58.353592 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:58.353600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:58.353881 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:58.354083 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:58.355883 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.357213 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.358455 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:58.359009 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:58.359114 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:58.361691 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:58.361701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:58.361714 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:58.361749 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:58.364682 (+ 2933us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:58.364947 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57610: Request ticket server kudu/127.5.117.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":437,"threads_started":1}
W20260929 15:58:58.365360 14922 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57610: Request ticket server kudu/127.5.117.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:58.365424 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14736: Not authorized: Client connection negotiation failed: client connection to 127.5.117.3:45549: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.117.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:58.365592 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14736
I20260929 15:58:58.384567 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14385
I20260929 15:58:58.386231 14447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:58.525988 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14385
2026-09-29T15:58:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21907 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T15:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:58Z Disabled control of system clock
I20260929 15:58:58.622066 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:43239
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:58:59.004032 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:59.004505 14927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:59.004719 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:59.012589 14927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:59.012734 14927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:59.012775 14927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:59.012903 14927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:59.013007 14927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:59.023145 14927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:43239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:59.024916 14927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:59.027087 14927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:59.040518 14932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:59.040467 14935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:59.040556 14933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:59.041226 14927 server_base.cc:1034] running on GCE node
I20260929 15:58:59.042039 14927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:59.043468 14927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:59.044677 14927 hybrid_clock.cc:648] HybridClock initialized: now 1790697539044629 us; error 41 us; skew 500 ppm
I20260929 15:58:59.045013 14927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:59.059649 14927 webserver.cc:460] Webserver started at http://127.5.117.62:44589/ using document root <none> and password file <none>
I20260929 15:58:59.060703 14927 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:59.060808 14927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:59.061142 14927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:59.064796 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "dfdb06c843bb40a48788a26e8039c710"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "bf9e4e757d3ecde4eb099e345a99332e"
server_key_iv: "3cddc6c100a5a32f54119f8cacd951e4"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.065691 14927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "dfdb06c843bb40a48788a26e8039c710"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "bf9e4e757d3ecde4eb099e345a99332e"
server_key_iv: "3cddc6c100a5a32f54119f8cacd951e4"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.072659 14927 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 15:58:59.077145 14941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:59.078929 14927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 15:58:59.079108 14927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "dfdb06c843bb40a48788a26e8039c710"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "bf9e4e757d3ecde4eb099e345a99332e"
server_key_iv: "3cddc6c100a5a32f54119f8cacd951e4"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.079339 14927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:59.097862 14927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:59.098842 14927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:59.099145 14927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:59.121138 14927 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:43239
I20260929 15:58:59.121160 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:43239 every 8 connection(s)
I20260929 15:58:59.123009 14927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:59.127825 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 14927
I20260929 15:58:59.128160 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:58:59.128925 5588 external_mini_cluster.cc:1428] Setting key 95b4645f5714e7cec123b41e70b31904
I20260929 15:58:59.129076 14993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:59.147552 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:59.131851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.5.117.62:43239)
0929 15:58:59.133332 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:59.133351 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:59.133400 (+ 49us) server_negotiation.cc:407] Connection header received
0929 15:58:59.135402 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:59.135445 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:59.136368 (+ 923us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:59.136896 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:59.137888 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.139437 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.141561 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.142379 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.142561 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:59.144250 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:59.144263 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:59.144285 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:59.145249 (+ 964us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:59.145369 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:59.145397 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:59.145520 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:59.145790 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 15:58:59.146290 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":598,"threads_started":1}
I20260929 15:58:59.147689 14993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710: Bootstrap starting.
I20260929 15:58:59.152359 14993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:59.153901 14993 log.cc:824] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:59.158010 14993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710: No bootstrap required, opened a new log
I20260929 15:58:59.167896 14993 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } }
I20260929 15:58:59.168663 14993 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:59.168949 14993 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdb06c843bb40a48788a26e8039c710, State: Initialized, Role: FOLLOWER
I20260929 15:58:59.170118 14993 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } }
I20260929 15:58:59.170465 14993 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:59.170634 14993 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:59.170838 14993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:59.173771 14993 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } }
I20260929 15:58:59.174373 14993 leader_election.cc:302] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfdb06c843bb40a48788a26e8039c710; no voters:
I20260929 15:58:59.175084 14993 leader_election.cc:288] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:59.175372 14998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:59.176354 14998 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [term 1 LEADER]: Becoming Leader. State: Replica: dfdb06c843bb40a48788a26e8039c710, State: Running, Role: LEADER
I20260929 15:58:59.177075 14998 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } }
I20260929 15:58:59.177584 14993 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:59.181262 14999 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfdb06c843bb40a48788a26e8039c710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } } }
I20260929 15:58:59.181676 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfdb06c843bb40a48788a26e8039c710. Latest consensus state: current_term: 1 leader_uuid: "dfdb06c843bb40a48788a26e8039c710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfdb06c843bb40a48788a26e8039c710" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43239 } } }
I20260929 15:58:59.182154 14999 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:59.182273 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:59.185276 15007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:59.191538 15007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:59.202234 15007 catalog_manager.cc:1347] Generated new cluster ID: bbab6b5bdb674a44945ae1a0b598b2b4
I20260929 15:58:59.202397 15007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:59.217979 15007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:59.219467 15007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:59.232628 15007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfdb06c843bb40a48788a26e8039c710: Generated new TSK 0
I20260929 15:58:59.233783 15007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:59.253117 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--builtin_ntp_servers=127.5.117.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 15:58:59.658574 15017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:59.658917 15017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:59.659080 15017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:59.666422 15017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:59.666787 15017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:59.667064 15017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:58:59.679102 15017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:58:59.680912 15017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:59.683225 15017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:59.699747 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:59.699819 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:59.699782 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:59.701079 15017 server_base.cc:1034] running on GCE node
I20260929 15:58:59.702035 15017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:59.703300 15017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:59.704574 15017 hybrid_clock.cc:648] HybridClock initialized: now 1790697539704518 us; error 44 us; skew 500 ppm
I20260929 15:58:59.704986 15017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:59.708185 15017 webserver.cc:460] Webserver started at http://127.5.117.1:44737/ using document root <none> and password file <none>
I20260929 15:58:59.709132 15017 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:59.709290 15017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:59.709693 15017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:59.713325 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1c5eb662677c48f4b855c76fa65647d0"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "3a4ec60bb84c11aabebfcb040ad4ef8c"
server_key_iv: "13f81c4065282bb2626e4d847b51abb0"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.714223 15017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1c5eb662677c48f4b855c76fa65647d0"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "3a4ec60bb84c11aabebfcb040ad4ef8c"
server_key_iv: "13f81c4065282bb2626e4d847b51abb0"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.720273 15017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 15:58:59.725634 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:59.727960 15017 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:59.728282 15017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1c5eb662677c48f4b855c76fa65647d0"
format_stamp: "Formatted at 2026-09-29 15:58:59 on dist-test-slave-b69v"
server_key: "3a4ec60bb84c11aabebfcb040ad4ef8c"
server_key_iv: "13f81c4065282bb2626e4d847b51abb0"
server_key_version: "encryptionkey@0"
I20260929 15:58:59.728650 15017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:59.762583 15017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:59.763634 15017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:59.763954 15017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:59.765210 15017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:59.767426 15017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:59.767557 15017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:59.767853 15017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:59.767992 15017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:59.806679 15017 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:39385
I20260929 15:58:59.806728 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:39385 every 8 connection(s)
I20260929 15:58:59.808195 15017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:59.817509 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15017
I20260929 15:58:59.817759 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:59.818491 5588 external_mini_cluster.cc:1428] Setting key 1064ec2192663b809495e12e20fec5a6
I20260929 15:58:59.822002 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--builtin_ntp_servers=127.5.117.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:58:59.823818 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:59.811236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:58933 (local address 127.5.117.62:43239)
0929 15:58:59.811791 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:59.811796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:59.812963 (+ 1167us) server_negotiation.cc:407] Connection header received
0929 15:58:59.814909 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:59.814916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:59.815027 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:59.815165 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:59.816369 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.817187 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.818982 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.819574 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.819717 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:59.822708 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:59.822719 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:59.822727 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:59.822755 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:59.822816 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:59.822847 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:59.822956 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:59.823193 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 15:58:59.823321 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":302,"threads_started":1}
I20260929 15:58:59.824643 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:59.811732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43239 (local address 127.5.117.1:58933)
0929 15:58:59.812765 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:59.812832 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:59.814610 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:59.815415 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:59.815423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:59.815509 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:59.816195 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:59.816214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.817372 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.817377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:59.818815 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:59.818829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:59.819833 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:59.819840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:59.820145 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:59.822368 (+ 2223us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:59.822371 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:59.822414 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:59.822497 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:59.822958 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:59.822966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:59.822982 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 15:58:59.823126 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 15:58:59.823428 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":366,"threads_started":1}
I20260929 15:58:59.826246 15144 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43239
I20260929 15:58:59.826570 15144 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:59.827467 15144 heartbeater.cc:511] Master 127.5.117.62:43239 requested a full tablet report, sending...
I20260929 15:58:59.829715 14958 ts_manager.cc:194] Registered new tserver with Master: 1c5eb662677c48f4b855c76fa65647d0 (127.5.117.1:39385)
I20260929 15:58:59.831321 14958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.1:58933
W20260929 15:59:00.205859 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:00.206208 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:00.206394 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:00.213256 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:00.213375 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:00.213557 15148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:59:00.222800 15148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:00.224676 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:00.226745 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:00.241082 15153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:00.241364 15154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:00.241890 15156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:00.242486 15148 server_base.cc:1034] running on GCE node
I20260929 15:59:00.243340 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:00.244359 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:00.245625 15148 hybrid_clock.cc:648] HybridClock initialized: now 1790697540245589 us; error 44 us; skew 500 ppm
I20260929 15:59:00.245988 15148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:00.248777 15148 webserver.cc:460] Webserver started at http://127.5.117.2:41717/ using document root <none> and password file <none>
I20260929 15:59:00.249513 15148 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:00.249650 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:00.250011 15148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:00.253187 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "db8c0d11309a4bb8a44d92c872ba4b5a"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "11c7d68229e827a86d71c291ba888dd3"
server_key_iv: "4ee0bf3cbd8ae02bc4aa18a67d9f23d3"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.254040 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "db8c0d11309a4bb8a44d92c872ba4b5a"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "11c7d68229e827a86d71c291ba888dd3"
server_key_iv: "4ee0bf3cbd8ae02bc4aa18a67d9f23d3"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.259305 15148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:59:00.263195 15162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:00.264894 15148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:59:00.265115 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "db8c0d11309a4bb8a44d92c872ba4b5a"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "11c7d68229e827a86d71c291ba888dd3"
server_key_iv: "4ee0bf3cbd8ae02bc4aa18a67d9f23d3"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.265372 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:00.281548 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:00.282671 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:00.283257 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:00.285096 15148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:00.286994 15148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:00.287177 15148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:00.287328 15148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:00.287422 15148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:00.326073 15148 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:46187
I20260929 15:59:00.326205 15274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:46187 every 8 connection(s)
I20260929 15:59:00.327665 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:59:00.335799 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15148
I20260929 15:59:00.336103 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:59:00.336608 5588 external_mini_cluster.cc:1428] Setting key 3bedfca803c20d82475be8bb90a2a7f9
I20260929 15:59:00.340828 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--builtin_ntp_servers=127.5.117.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:59:00.343710 15277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:00.331005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:43387 (local address 127.5.117.62:43239)
0929 15:59:00.331520 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:00.331525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:00.332965 (+ 1440us) server_negotiation.cc:407] Connection header received
0929 15:59:00.334955 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:00.334962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:00.335072 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:00.335208 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:00.336556 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:00.337676 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:00.340124 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:00.340927 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:00.341055 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:00.342790 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:00.342796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:00.342800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:00.342820 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:00.342864 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:00.342876 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:00.342974 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:00.343221 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 15:59:00.343332 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":291,"threads_started":1}
I20260929 15:59:00.344944 15278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:00.331576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43239 (local address 127.5.117.2:43387)
0929 15:59:00.332755 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:00.332818 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:00.334659 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:00.335467 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:00.335476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:00.335570 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:00.336374 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:00.336393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:00.337884 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:00.337891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:00.339897 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:00.339912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:00.341357 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:00.341367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:00.341617 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:00.342431 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:59:00.342434 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:00.342478 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:59:00.342568 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:00.342972 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:00.342985 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:00.342996 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:59:00.343209 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 15:59:00.343632 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":473,"threads_started":1}
I20260929 15:59:00.346429 15275 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43239
I20260929 15:59:00.346781 15275 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:00.347782 15275 heartbeater.cc:511] Master 127.5.117.62:43239 requested a full tablet report, sending...
I20260929 15:59:00.349539 14958 ts_manager.cc:194] Registered new tserver with Master: db8c0d11309a4bb8a44d92c872ba4b5a (127.5.117.2:46187)
I20260929 15:59:00.350418 14958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.2:43387
W20260929 15:59:00.784531 15279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:00.785261 15279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:00.785594 15279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:00.803907 15279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:00.804210 15279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:00.804551 15279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:59:00.824620 15279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:00.827775 15279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:00.831629 15279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:59:00.834723 15144 heartbeater.cc:503] Master 127.5.117.62:43239 was elected leader, sending a full tablet report...
W20260929 15:59:00.860973 15284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:00.861022 15285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:00.861943 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:00.863519 15279 server_base.cc:1034] running on GCE node
I20260929 15:59:00.864992 15279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:00.866842 15279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:00.870522 15279 hybrid_clock.cc:648] HybridClock initialized: now 1790697540870339 us; error 87 us; skew 500 ppm
I20260929 15:59:00.871331 15279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:00.877072 15279 webserver.cc:460] Webserver started at http://127.5.117.3:37109/ using document root <none> and password file <none>
I20260929 15:59:00.879186 15279 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:00.879453 15279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:00.880235 15279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:00.886744 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "de1283a6482a43fcb034f202536a6921"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "75f3fe2874420f7e390b0f6c562db73d"
server_key_iv: "932f142a5f67a6cc751127cd8b3515e1"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.888446 15279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "de1283a6482a43fcb034f202536a6921"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "75f3fe2874420f7e390b0f6c562db73d"
server_key_iv: "932f142a5f67a6cc751127cd8b3515e1"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.902176 15279 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260929 15:59:00.910383 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:00.913561 15279 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260929 15:59:00.913859 15279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "de1283a6482a43fcb034f202536a6921"
format_stamp: "Formatted at 2026-09-29 15:59:00 on dist-test-slave-b69v"
server_key: "75f3fe2874420f7e390b0f6c562db73d"
server_key_iv: "932f142a5f67a6cc751127cd8b3515e1"
server_key_version: "encryptionkey@0"
I20260929 15:59:00.914280 15279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:01.018886 15279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:01.021903 15279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:01.022748 15279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:01.025309 15279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:01.029973 15279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:01.030364 15279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:01.030699 15279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:01.030958 15279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:01.122198 15279 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:36677
I20260929 15:59:01.122318 15405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:36677 every 8 connection(s)
I20260929 15:59:01.126132 15279 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:59:01.130923 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15279
I20260929 15:59:01.131486 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:59:01.132829 5588 external_mini_cluster.cc:1428] Setting key 5fd9d4025e682554132125467c079d17
I20260929 15:59:01.163237 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:01.134023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:34543 (local address 127.5.117.62:43239)
0929 15:59:01.135452 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:01.135481 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:01.140459 (+ 4978us) server_negotiation.cc:407] Connection header received
0929 15:59:01.145865 (+ 5406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:01.145883 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:01.146174 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:01.146587 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:01.150043 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.151990 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.156814 (+ 4824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.157812 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.158066 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:01.161342 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:01.161349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:01.161352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:01.161412 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:01.161555 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:01.161594 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:01.161772 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:01.162223 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 15:59:01.162446 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":629,"threads_started":1}
I20260929 15:59:01.166059 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:01.136133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43239 (local address 127.5.117.3:34543)
0929 15:59:01.139995 (+ 3862us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:01.140186 (+ 191us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:01.144978 (+ 4792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:01.147265 (+ 2287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:01.147286 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:01.147575 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:01.149618 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:01.149672 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.152416 (+ 2744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.152433 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:01.156568 (+ 4135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:01.156598 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.158232 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.158245 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:01.158757 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:01.160544 (+ 1787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:59:01.160547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:01.160677 (+ 130us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:59:01.160919 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:01.161765 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:01.161787 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:01.161805 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 15:59:01.162193 (+ 388us) client_negotiation.cc:240] Negotiation successful
0929 15:59:01.162984 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1986,"thread_start_us":1418,"threads_started":1}
I20260929 15:59:01.169616 15406 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43239
I20260929 15:59:01.170404 15406 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:01.172752 15406 heartbeater.cc:511] Master 127.5.117.62:43239 requested a full tablet report, sending...
I20260929 15:59:01.176569 14958 ts_manager.cc:194] Registered new tserver with Master: de1283a6482a43fcb034f202536a6921 (127.5.117.3:36677)
I20260929 15:59:01.178886 14958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.3:34543
I20260929 15:59:01.188184 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:59:01.208463 15408 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:59:01.203552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.5.117.62:43239)
0929 15:59:01.204144 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:01.204178 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:01.206052 (+ 1874us) server_negotiation.cc:407] Connection header received
0929 15:59:01.206296 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:01.206312 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:01.206465 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:01.206766 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:01.207805 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42498: BlockingRecv error: recv got EOF from 127.0.0.1:42498 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260929 15:59:01.210263 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15017
I20260929 15:59:01.239656 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:01.213280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.5.117.1:39385)
0929 15:59:01.215585 (+ 2305us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:01.215596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:01.215689 (+ 93us) server_negotiation.cc:407] Connection header received
0929 15:59:01.215939 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:01.215964 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:01.216929 (+ 965us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:01.217438 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:01.218999 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.222031 (+ 3032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.227661 (+ 5630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.229978 (+ 2317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.230376 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:01.234367 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:01.234402 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:01.234454 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:01.236447 (+ 1993us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:01.236631 (+ 184us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:01.236698 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:01.236967 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:01.237478 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 15:59:01.237856 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":1067,"threads_started":1}
I20260929 15:59:01.243736 15139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:01.353893 15275 heartbeater.cc:503] Master 127.5.117.62:43239 was elected leader, sending a full tablet report...
I20260929 15:59:01.440524 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15017
I20260929 15:59:01.490136 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15148
I20260929 15:59:01.516247 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:01.492737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.5.117.2:46187)
0929 15:59:01.494286 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:01.494313 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:01.494400 (+ 87us) server_negotiation.cc:407] Connection header received
0929 15:59:01.494566 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:01.494591 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:01.495285 (+ 694us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:01.495775 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:01.496913 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.499602 (+ 2689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.503620 (+ 4018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.506282 (+ 2662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.507427 (+ 1145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:01.511201 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:01.511230 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:01.511281 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:01.513395 (+ 2114us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:01.513581 (+ 186us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:01.513647 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:01.514030 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:01.514496 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 15:59:01.515083 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":779,"threads_started":1}
I20260929 15:59:01.520864 15270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:01.700887 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15148
I20260929 15:59:01.739908 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15279
I20260929 15:59:01.760155 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:01.742160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.5.117.3:36677)
0929 15:59:01.743208 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:01.743217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:01.743284 (+ 67us) server_negotiation.cc:407] Connection header received
0929 15:59:01.743462 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:01.743480 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:01.744229 (+ 749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:01.744811 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:01.745809 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.748254 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.751232 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:01.752580 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:01.753012 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:01.755911 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:01.755932 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:01.755974 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:01.757783 (+ 1809us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:01.757867 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:01.757900 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:01.758237 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:01.758727 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 15:59:01.759167 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":523,"threads_started":1}
I20260929 15:59:01.763813 15401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:01.926405 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15279
I20260929 15:59:01.961935 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 14927
I20260929 15:59:01.965880 14988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:02.174247 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 14927
2026-09-29T15:59:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3673 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T15:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:59:02Z Disabled control of system clock
I20260929 15:59:02.360031 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:43069
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:43069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:59:02.910669 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:02.910983 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:02.911128 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:02.918501 15426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:59:02.918602 15426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:02.918676 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:02.918766 15426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:59:02.918808 15426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:59:02.927770 15426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:43069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:02.929350 15426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:02.931479 15426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:02.945366 15431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:02.945422 15432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:02.945292 15434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:02.946473 15426 server_base.cc:1034] running on GCE node
I20260929 15:59:02.947398 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:02.949276 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:02.950521 15426 hybrid_clock.cc:648] HybridClock initialized: now 1790697542950478 us; error 36 us; skew 500 ppm
I20260929 15:59:02.951001 15426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:02.953817 15426 webserver.cc:460] Webserver started at http://127.5.117.62:36345/ using document root <none> and password file <none>
I20260929 15:59:02.954667 15426 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:02.954764 15426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:02.955163 15426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:02.958915 15426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "cad599be2f6d43bb8e1bb0689280d534"
format_stamp: "Formatted at 2026-09-29 15:59:02 on dist-test-slave-b69v"
server_key: "27a9582ae67f88238c6dd35c69425ebb"
server_key_iv: "323ad0c467f36ca2e9ad11ed5f82235b"
server_key_version: "encryptionkey@0"
I20260929 15:59:02.959897 15426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "cad599be2f6d43bb8e1bb0689280d534"
format_stamp: "Formatted at 2026-09-29 15:59:02 on dist-test-slave-b69v"
server_key: "27a9582ae67f88238c6dd35c69425ebb"
server_key_iv: "323ad0c467f36ca2e9ad11ed5f82235b"
server_key_version: "encryptionkey@0"
I20260929 15:59:02.967358 15426 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260929 15:59:02.972393 15440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:02.974025 15426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:59:02.974243 15426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "cad599be2f6d43bb8e1bb0689280d534"
format_stamp: "Formatted at 2026-09-29 15:59:02 on dist-test-slave-b69v"
server_key: "27a9582ae67f88238c6dd35c69425ebb"
server_key_iv: "323ad0c467f36ca2e9ad11ed5f82235b"
server_key_version: "encryptionkey@0"
I20260929 15:59:02.974469 15426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:02.981261 15426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:02.982139 15426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:02.982470 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:03.005327 15491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:43069 every 8 connection(s)
I20260929 15:59:03.005432 15426 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:43069
I20260929 15:59:03.007397 15426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:59:03.011940 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:03.013230 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15426
I20260929 15:59:03.013481 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:59:03.014514 5588 external_mini_cluster.cc:1428] Setting key 0d837200cc55a209a647f97643687491
I20260929 15:59:03.024912 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:03.017163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.5.117.62:43069)
0929 15:59:03.018455 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:03.018468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:03.018518 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:59:03.020275 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:03.020299 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:03.020780 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:03.021158 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:03.022424 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:03.022432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:03.022444 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:03.022459 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:03.022530 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:03.022546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:03.022716 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:03.022886 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 15:59:03.023540 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":498,"threads_started":1}
I20260929 15:59:03.027928 15492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534: Bootstrap starting.
I20260929 15:59:03.031265 15492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:03.032450 15492 log.cc:824] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:03.035959 15492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534: No bootstrap required, opened a new log
I20260929 15:59:03.044551 15492 raft_consensus.cc:348] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } }
I20260929 15:59:03.044988 15492 raft_consensus.cc:374] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:03.045113 15492 raft_consensus.cc:729] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cad599be2f6d43bb8e1bb0689280d534, State: Initialized, Role: FOLLOWER
I20260929 15:59:03.045719 15492 consensus_queue.cc:260] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } }
I20260929 15:59:03.046005 15492 raft_consensus.cc:388] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:59:03.046175 15492 raft_consensus.cc:482] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:59:03.046371 15492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:03.048794 15492 raft_consensus.cc:504] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } }
I20260929 15:59:03.049361 15492 leader_election.cc:302] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cad599be2f6d43bb8e1bb0689280d534; no voters:
I20260929 15:59:03.050231 15492 leader_election.cc:288] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:59:03.050518 15497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:03.051438 15497 raft_consensus.cc:686] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [term 1 LEADER]: Becoming Leader. State: Replica: cad599be2f6d43bb8e1bb0689280d534, State: Running, Role: LEADER
I20260929 15:59:03.052228 15497 consensus_queue.cc:237] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } }
I20260929 15:59:03.052716 15492 sys_catalog.cc:564] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:59:03.055711 15499 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cad599be2f6d43bb8e1bb0689280d534. Latest consensus state: current_term: 1 leader_uuid: "cad599be2f6d43bb8e1bb0689280d534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } } }
I20260929 15:59:03.056170 15499 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:03.055858 15498 sys_catalog.cc:455] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cad599be2f6d43bb8e1bb0689280d534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cad599be2f6d43bb8e1bb0689280d534" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 43069 } } }
I20260929 15:59:03.056990 15498 sys_catalog.cc:458] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:03.057828 15505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:59:03.065070 15505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:59:03.075369 15505 catalog_manager.cc:1347] Generated new cluster ID: dbd66584ea514b7f8cd6cfec5717e418
I20260929 15:59:03.075539 15505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:59:03.088865 15505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:59:03.089388 15505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:59:03.097704 15505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cad599be2f6d43bb8e1bb0689280d534: Generated new TSK 0
I20260929 15:59:03.098721 15505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:59:03.111161 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--builtin_ntp_servers=127.5.117.20:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:59:03.508118 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:03.508504 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:03.508659 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:03.516525 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:03.516665 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:03.516868 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:59:03.527032 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:03.528782 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:03.531550 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:03.547171 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:03.547222 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:03.547833 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:03.548252 15516 server_base.cc:1034] running on GCE node
I20260929 15:59:03.548856 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:03.549852 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:03.551110 15516 hybrid_clock.cc:648] HybridClock initialized: now 1790697543551053 us; error 59 us; skew 500 ppm
I20260929 15:59:03.551478 15516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:03.554222 15516 webserver.cc:460] Webserver started at http://127.5.117.1:42525/ using document root <none> and password file <none>
I20260929 15:59:03.555020 15516 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:03.555164 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:03.555533 15516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:03.559098 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1a3e1eaa24914469ab7dd98aa7e36e0e"
format_stamp: "Formatted at 2026-09-29 15:59:03 on dist-test-slave-b69v"
server_key: "9fc682f64e211592a4cf3e647dc16286"
server_key_iv: "49b344c8889d0a846e3417fa53c01889"
server_key_version: "encryptionkey@0"
I20260929 15:59:03.559952 15516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1a3e1eaa24914469ab7dd98aa7e36e0e"
format_stamp: "Formatted at 2026-09-29 15:59:03 on dist-test-slave-b69v"
server_key: "9fc682f64e211592a4cf3e647dc16286"
server_key_iv: "49b344c8889d0a846e3417fa53c01889"
server_key_version: "encryptionkey@0"
I20260929 15:59:03.565418 15516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:59:03.569851 15530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:03.571852 15516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:59:03.572268 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1a3e1eaa24914469ab7dd98aa7e36e0e"
format_stamp: "Formatted at 2026-09-29 15:59:03 on dist-test-slave-b69v"
server_key: "9fc682f64e211592a4cf3e647dc16286"
server_key_iv: "49b344c8889d0a846e3417fa53c01889"
server_key_version: "encryptionkey@0"
I20260929 15:59:03.572525 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:03.578619 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:03.579697 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:03.580003 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:03.581080 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:03.582561 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:03.582700 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:03.582857 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:03.582962 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:03.618544 15516 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:34741
I20260929 15:59:03.618631 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:34741 every 8 connection(s)
I20260929 15:59:03.619936 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:59:03.626312 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15516
I20260929 15:59:03.626570 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:59:03.627383 5588 external_mini_cluster.cc:1428] Setting key b5eca8dc640b3fb88ee5144e57eb48ac
I20260929 15:59:03.628774 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:03.622695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:60251 (local address 127.5.117.62:43069)
0929 15:59:03.623192 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:03.623200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:03.624447 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 15:59:03.626096 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:03.626104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:03.626183 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:03.626325 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:03.627887 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:03.627893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:03.627897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:03.627916 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:03.627970 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:03.627987 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:03.628145 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:03.628396 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 15:59:03.628464 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":231,"threads_started":1}
I20260929 15:59:03.629871 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:03.623303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43069 (local address 127.5.117.1:60251)
0929 15:59:03.624295 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:03.624341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:03.625827 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:03.626623 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:03.626638 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:03.626745 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:03.627575 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:59:03.627577 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:03.627629 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:59:03.627710 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:03.628122 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:03.628133 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:03.628142 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 15:59:03.628446 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 15:59:03.628683 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
I20260929 15:59:03.630831 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--builtin_ntp_servers=127.5.117.20:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:59:03.631368 15643 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43069
I20260929 15:59:03.631683 15643 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:03.632382 15643 heartbeater.cc:511] Master 127.5.117.62:43069 requested a full tablet report, sending...
I20260929 15:59:03.634778 15457 ts_manager.cc:194] Registered new tserver with Master: 1a3e1eaa24914469ab7dd98aa7e36e0e (127.5.117.1:34741)
W20260929 15:59:04.045810 15647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:04.046176 15647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:04.046370 15647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:04.057164 15647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:04.057307 15647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:04.057504 15647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:59:04.069139 15647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:04.070748 15647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:04.073004 15647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:04.089452 15653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:04.089617 15655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:04.089484 15652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:04.091285 15647 server_base.cc:1034] running on GCE node
I20260929 15:59:04.092437 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:04.093816 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:04.095115 15647 hybrid_clock.cc:648] HybridClock initialized: now 1790697544095032 us; error 79 us; skew 500 ppm
I20260929 15:59:04.095585 15647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:04.105024 15647 webserver.cc:460] Webserver started at http://127.5.117.2:36223/ using document root <none> and password file <none>
I20260929 15:59:04.106036 15647 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:04.106156 15647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:04.106585 15647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:04.110334 15647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "95592970dcc84104b1640263559f8c30"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "a9a3a0f304514d74dcafe765ee36cb14"
server_key_iv: "be020b30a872608e5c14a06f83198ac4"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.111325 15647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "95592970dcc84104b1640263559f8c30"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "a9a3a0f304514d74dcafe765ee36cb14"
server_key_iv: "be020b30a872608e5c14a06f83198ac4"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.118491 15647 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 15:59:04.123163 15661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.125018 15647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:59:04.125248 15647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "95592970dcc84104b1640263559f8c30"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "a9a3a0f304514d74dcafe765ee36cb14"
server_key_iv: "be020b30a872608e5c14a06f83198ac4"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.125563 15647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:04.132860 15647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:04.133957 15647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:04.134308 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:04.135439 15647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:04.137346 15647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:04.137444 15647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.137626 15647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:04.137722 15647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.177888 15647 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:45181
I20260929 15:59:04.177940 15773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:45181 every 8 connection(s)
I20260929 15:59:04.179513 15647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:59:04.185061 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15647
I20260929 15:59:04.185410 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:59:04.186039 5588 external_mini_cluster.cc:1428] Setting key 83898ad92e7b675ef685cd4fc41ce13e
I20260929 15:59:04.189600 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--builtin_ntp_servers=127.5.117.20:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:59:04.190680 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.182527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:41267 (local address 127.5.117.62:43069)
0929 15:59:04.183087 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.183094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.185530 (+ 2436us) server_negotiation.cc:407] Connection header received
0929 15:59:04.187845 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.187851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.187941 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.188136 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.189836 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:04.189845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:04.189853 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:04.189875 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:04.189936 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:04.189954 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:04.190085 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:04.190347 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 15:59:04.190405 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
I20260929 15:59:04.192171 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.183195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43069 (local address 127.5.117.2:41267)
0929 15:59:04.185212 (+ 2017us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:04.185301 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:04.187464 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:04.188425 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:04.188439 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:04.188587 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:04.189405 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:59:04.189410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:04.189466 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:59:04.189604 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:04.190094 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:04.190107 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:04.190118 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:59:04.190443 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 15:59:04.190756 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":471,"threads_started":1}
I20260929 15:59:04.193790 15774 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43069
I20260929 15:59:04.194283 15774 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:04.195006 15774 heartbeater.cc:511] Master 127.5.117.62:43069 requested a full tablet report, sending...
I20260929 15:59:04.196686 15457 ts_manager.cc:194] Registered new tserver with Master: 95592970dcc84104b1640263559f8c30 (127.5.117.2:45181)
W20260929 15:59:04.566409 15778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:04.566702 15778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:04.566870 15778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:04.574290 15778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:04.574380 15778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:04.574555 15778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:59:04.583706 15778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:43069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:04.585160 15778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:04.587149 15778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:04.601315 15783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:04.601285 15784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:04.601552 15786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:04.602146 15778 server_base.cc:1034] running on GCE node
I20260929 15:59:04.602731 15778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:04.603809 15778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:04.605041 15778 hybrid_clock.cc:648] HybridClock initialized: now 1790697544605002 us; error 57 us; skew 500 ppm
I20260929 15:59:04.605406 15778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:59:04.608853 15778 webserver.cc:460] Webserver started at http://127.5.117.3:34969/ using document root <none> and password file <none>
I20260929 15:59:04.609695 15778 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:04.609789 15778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:04.610189 15778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:04.614143 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "340e3586ee4b4724ba10de77f6c80a99"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "5b8d64be1b76d8fb2d94e74acfc433c7"
server_key_iv: "2ada7e62c4aae8cc9b4abab92e3b0a04"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.615018 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "340e3586ee4b4724ba10de77f6c80a99"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "5b8d64be1b76d8fb2d94e74acfc433c7"
server_key_iv: "2ada7e62c4aae8cc9b4abab92e3b0a04"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.621212 15778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 15:59:04.625160 15792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.626895 15778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:59:04.627103 15778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "340e3586ee4b4724ba10de77f6c80a99"
format_stamp: "Formatted at 2026-09-29 15:59:04 on dist-test-slave-b69v"
server_key: "5b8d64be1b76d8fb2d94e74acfc433c7"
server_key_iv: "2ada7e62c4aae8cc9b4abab92e3b0a04"
server_key_version: "encryptionkey@0"
I20260929 15:59:04.627353 15778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:04.634195 15778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:04.635134 15778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:04.635444 15778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:04.636543 15778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:04.638106 15643 heartbeater.cc:503] Master 127.5.117.62:43069 was elected leader, sending a full tablet report...
I20260929 15:59:04.638304 15778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:04.638419 15778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.638542 15778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:04.638623 15778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:04.677145 15778 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:33971
I20260929 15:59:04.677196 15904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:33971 every 8 connection(s)
I20260929 15:59:04.678730 15778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:59:04.681572 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15778
I20260929 15:59:04.681780 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:59:04.682337 5588 external_mini_cluster.cc:1428] Setting key 71a74e94315cf2d107becd60e5ee19ed
I20260929 15:59:04.688156 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.681854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:50927 (local address 127.5.117.62:43069)
0929 15:59:04.682102 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.682109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.683976 (+ 1867us) server_negotiation.cc:407] Connection header received
0929 15:59:04.685925 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.685930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.685990 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.686073 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.687478 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:04.687490 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:04.687493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:04.687509 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:04.687544 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:04.687556 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:04.687649 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:04.687807 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 15:59:04.687862 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:59:04.689198 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.682607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:43069 (local address 127.5.117.3:50927)
0929 15:59:04.683749 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:04.683835 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:04.685639 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:04.686266 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:04.686273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:04.686370 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:04.687176 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:59:04.687178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:04.687219 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:59:04.687311 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:04.687630 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:04.687636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:04.687641 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:59:04.687859 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 15:59:04.688141 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":392,"threads_started":1}
I20260929 15:59:04.690567 15905 heartbeater.cc:347] Connected to a master server at 127.5.117.62:43069
I20260929 15:59:04.690886 15905 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:04.691543 15905 heartbeater.cc:511] Master 127.5.117.62:43069 requested a full tablet report, sending...
I20260929 15:59:04.693151 15457 ts_manager.cc:194] Registered new tserver with Master: 340e3586ee4b4724ba10de77f6c80a99 (127.5.117.3:33971)
I20260929 15:59:04.698170 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:59:04.704923 15776 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:59:04.703201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.5.117.62:43069)
0929 15:59:04.703434 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.703443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.703915 (+ 472us) server_negotiation.cc:407] Connection header received
0929 15:59:04.704099 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.704108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.704162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.704242 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.704668 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50422: BlockingRecv error: recv got EOF from 127.0.0.1:50422 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:59:04.705382 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15516
I20260929 15:59:04.709647 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.706594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.5.117.1:34741)
0929 15:59:04.707172 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.707180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.707206 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:59:04.707273 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.707279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.707483 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.707672 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.708861 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:04.708868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:04.708879 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:04.708891 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:04.708933 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:04.708947 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:04.709161 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:04.709231 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 15:59:04.709351 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":287,"threads_started":1}
I20260929 15:59:04.710902 15638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:04.797073 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15516
I20260929 15:59:04.817272 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15647
I20260929 15:59:04.821663 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.818432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.5.117.2:45181)
0929 15:59:04.819076 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.819085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.819115 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:59:04.819195 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.819203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.819512 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.819665 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.820831 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:04.820839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:04.820852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:04.820866 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:04.820913 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:04.820928 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:04.821046 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:04.821207 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 15:59:04.821301 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":334,"threads_started":1}
I20260929 15:59:04.823127 15769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:04.911649 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15647
I20260929 15:59:04.931164 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15778
I20260929 15:59:04.936365 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:04.932653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.5.117.3:33971)
0929 15:59:04.933463 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:04.933471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:04.933510 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:59:04.933593 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:04.933600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:04.933883 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:04.934115 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:04.935446 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:04.935453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:04.935466 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:59:04.935483 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:04.935527 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:59:04.935542 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:04.935765 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:04.935848 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 15:59:04.935973 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":420,"threads_started":1}
I20260929 15:59:04.937809 15900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:05.029371 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15778
I20260929 15:59:05.047991 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15426
I20260929 15:59:05.049712 15487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:05.164718 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15426
2026-09-29T15:59:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2959 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:59:05 dist-test-slave-b69v krb5kdc[15923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:59:05 dist-test-slave-b69v krb5kdc[15923](info): set up 2 sockets
Sep 29 15:59:05 dist-test-slave-b69v krb5kdc[15923](info): commencing operation
krb5kdc: starting...
W20260929 15:59:06.313669 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:59:06 dist-test-slave-b69v krb5kdc[15923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:59:06.572808 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41295
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:41295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:59:06.957365 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:06.957834 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:06.958027 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:06.966096 15936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:59:06.966281 15936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:06.966398 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:06.966507 15936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:59:06.966600 15936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:59:06.977507 15936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:41295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:06.979080 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:06.980706 15936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:06.994040 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:06.994155 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:06.994123 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:06.994846 15936 server_base.cc:1034] running on GCE node
I20260929 15:59:06.995631 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:06.997197 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:06.998617 15936 hybrid_clock.cc:648] HybridClock initialized: now 1790697546998514 us; error 66 us; skew 500 ppm
I20260929 15:59:06.999298 15936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:06 dist-test-slave-b69v krb5kdc[15923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:07.003338 15936 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:59:07.005414 15936 webserver.cc:460] Webserver started at http://127.5.117.62:46351/ using document root <none> and password file <none>
I20260929 15:59:07.006306 15936 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:07.006496 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:07.006858 15936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:07.010411 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "1cf5b753901349daa578e65909ca0cb1"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "05f9da4ed39536b8dc618af87195e575"
server_key_iv: "a95b359237bf184e41fcd2a2bb918021"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.011251 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "1cf5b753901349daa578e65909ca0cb1"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "05f9da4ed39536b8dc618af87195e575"
server_key_iv: "a95b359237bf184e41fcd2a2bb918021"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.017122 15936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 15:59:07.021068 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:07.022653 15936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:59:07.022867 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "1cf5b753901349daa578e65909ca0cb1"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "05f9da4ed39536b8dc618af87195e575"
server_key_iv: "a95b359237bf184e41fcd2a2bb918021"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.023104 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:07.048964 15936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:07.054409 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:07.054761 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:07.078011 15936 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:41295
I20260929 15:59:07.078032 16002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:41295 every 8 connection(s)
I20260929 15:59:07.079654 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:59:07.080701 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 15936
I20260929 15:59:07.080971 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:59:07.081871 5588 external_mini_cluster.cc:1428] Setting key 2fd3f064f9bf1c92f64ba0d25bbfcf5f
I20260929 15:59:07.085250 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:59:07 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:59:07.096572 16003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: Bootstrap starting.
I20260929 15:59:07.101397 16003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:07.102907 16003 log.cc:824] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:07.107138 16003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: No bootstrap required, opened a new log
I20260929 15:59:07.107180 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:07.084390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.5.117.62:41295)
0929 15:59:07.085197 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:07.085211 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:07.085258 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:59:07.087054 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:07.087080 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:07.087473 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:07.087924 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:07.089575 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.090965 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.092336 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.093020 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.093235 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:07.097513 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:07.097523 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:07.097543 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:07.097587 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:07.101992 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:07.102862 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:07.102871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:07.102881 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:07.103119 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:07.103552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:07.103561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:07.103570 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:07.104419 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:07.104778 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:07.105189 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 15:59:07.105667 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260929 15:59:07.116662 16003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } }
I20260929 15:59:07.117051 16003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:07.117131 16003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf5b753901349daa578e65909ca0cb1, State: Initialized, Role: FOLLOWER
I20260929 15:59:07.117760 16003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } }
I20260929 15:59:07.117990 16003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:59:07.118096 16003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:59:07.118225 16003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:07.120749 16003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } }
I20260929 15:59:07.121405 16003 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf5b753901349daa578e65909ca0cb1; no voters:
I20260929 15:59:07.122116 16003 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:59:07.122269 16008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:07.123613 16008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf5b753901349daa578e65909ca0cb1, State: Running, Role: LEADER
I20260929 15:59:07.124253 16008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } }
I20260929 15:59:07.124818 16003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:59:07.127125 16010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf5b753901349daa578e65909ca0cb1. Latest consensus state: current_term: 1 leader_uuid: "1cf5b753901349daa578e65909ca0cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } } }
I20260929 15:59:07.127488 16010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:07.128129 16009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf5b753901349daa578e65909ca0cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf5b753901349daa578e65909ca0cb1" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41295 } } }
I20260929 15:59:07.128582 16009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:07.132783 16014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:59:07.138525 16014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:59:07.148392 16014 catalog_manager.cc:1347] Generated new cluster ID: cbf0bb0b63ce4591a72fd54d6584d9d6
I20260929 15:59:07.148518 16014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:59:07.169982 16014 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:59:07.171310 16014 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:59:07.177627 16014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: Generated new TSK 0
I20260929 15:59:07.178607 16014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:59:07.245359 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:59:07.621436 16030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:07.621750 16030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:07.621852 16030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:07.629386 16030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:07.629490 16030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:07.629624 16030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 15:59:07.639629 16030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:07.640924 16030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:07.642385 16030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:07.656558 16035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:07.656662 16038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:07.656659 16036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:07.663641 16030 server_base.cc:1034] running on GCE node
I20260929 15:59:07.664461 16030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:07.665560 16030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:07.666795 16030 hybrid_clock.cc:648] HybridClock initialized: now 1790697547666748 us; error 37 us; skew 500 ppm
I20260929 15:59:07.667199 16030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:07 dist-test-slave-b69v krb5kdc[15923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:07.671154 16030 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 15:59:07.673117 16030 webserver.cc:460] Webserver started at http://127.5.117.1:42975/ using document root <none> and password file <none>
I20260929 15:59:07.673967 16030 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:07.674124 16030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:07.674487 16030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:07.678010 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "71135f09d8b84db392659072830ae3e4"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "1d0f06925e1c44289de68aea18a23c92"
server_key_iv: "571521faba6cf702599a5c1dc2911159"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.678898 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "71135f09d8b84db392659072830ae3e4"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "1d0f06925e1c44289de68aea18a23c92"
server_key_iv: "571521faba6cf702599a5c1dc2911159"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.684610 16030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 15:59:07.688194 16045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:07.689551 16030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:59:07.689743 16030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "71135f09d8b84db392659072830ae3e4"
format_stamp: "Formatted at 2026-09-29 15:59:07 on dist-test-slave-b69v"
server_key: "1d0f06925e1c44289de68aea18a23c92"
server_key_iv: "571521faba6cf702599a5c1dc2911159"
server_key_version: "encryptionkey@0"
I20260929 15:59:07.689957 16030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:07.713330 16030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:07.717422 16030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:07.717754 16030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:07.718752 16030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:07.720199 16030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:07.720328 16030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:07.720496 16030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:07.720579 16030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:07.760723 16030 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:40873
I20260929 15:59:07.760774 16157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:40873 every 8 connection(s)
I20260929 15:59:07.762174 16030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 15:59:07.764142 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 16030
I20260929 15:59:07.764374 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 15:59:07.765375 5588 external_mini_cluster.cc:1428] Setting key 37252cb874366e02b7cca0c0328816b8
Sep 29 15:59:07 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697547, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:59:07.787365 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:07.765567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:53263 (local address 127.5.117.62:41295)
0929 15:59:07.766262 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:07.766267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:07.767254 (+ 987us) server_negotiation.cc:407] Connection header received
0929 15:59:07.769471 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:07.769481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:07.769579 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:07.769716 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:07.771352 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.772515 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.774156 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.774881 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.775031 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:07.779228 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:07.779238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:07.779247 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:07.779300 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:07.782422 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:07.783740 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:07.783747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:07.783750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:07.784024 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:07.784811 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:07.784815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:07.784818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:07.785491 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:07.785649 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:07.786776 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0929 15:59:07.787046 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":336,"threads_started":1}
I20260929 15:59:07.789145 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:07.766088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41295 (local address 127.5.117.1:53263)
0929 15:59:07.767076 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:07.767127 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:07.769175 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:07.769959 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:07.769973 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:07.770567 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:07.771166 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:07.771183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.772679 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.772683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:07.773989 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:07.774001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:07.775121 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:07.775127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:07.775345 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:07.776212 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:07.776236 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:07.778948 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:07.782587 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:07.782598 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:07.782620 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:07.783415 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:07.784356 (+ 941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:07.784363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:07.784376 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:07.784654 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:07.785684 (+ 1030us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:07.785695 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:07.786228 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 15:59:07.786595 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 15:59:07.787268 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":439,"threads_started":1}
I20260929 15:59:07.791122 16158 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41295
I20260929 15:59:07.791496 16158 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:07.792498 16158 heartbeater.cc:511] Master 127.5.117.62:41295 requested a full tablet report, sending...
I20260929 15:59:07.795172 15968 ts_manager.cc:194] Registered new tserver with Master: 71135f09d8b84db392659072830ae3e4 (127.5.117.1:40873)
I20260929 15:59:07.796955 15968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:53263
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 15:59:07.840157 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:59:08.215971 16164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:08.216358 16164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:08.216475 16164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:08.223595 16164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:08.223691 16164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:08.223807 16164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 15:59:08.234208 16164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:08.236274 16164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:08.238515 16164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:08.258281 16172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:08.258420 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:08.258404 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:08.262050 16164 server_base.cc:1034] running on GCE node
I20260929 15:59:08.262815 16164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:08.264228 16164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:08.265465 16164 hybrid_clock.cc:648] HybridClock initialized: now 1790697548265392 us; error 55 us; skew 500 ppm
I20260929 15:59:08.265826 16164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:08 dist-test-slave-b69v krb5kdc[15923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697548, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:08.269569 16164 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 15:59:08.271384 16164 webserver.cc:460] Webserver started at http://127.5.117.2:38409/ using document root <none> and password file <none>
I20260929 15:59:08.272367 16164 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:08.272519 16164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:08.272895 16164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:08.276396 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "45ed9b933b144bed8c8ae0f35181307e"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "20db777704827a4852bb273f66b728eb"
server_key_iv: "16136d1305e86143f351f0e7dc3b966f"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.277262 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "45ed9b933b144bed8c8ae0f35181307e"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "20db777704827a4852bb273f66b728eb"
server_key_iv: "16136d1305e86143f351f0e7dc3b966f"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.283250 16164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:59:08.287256 16179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.289249 16164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 15:59:08.289439 16164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "45ed9b933b144bed8c8ae0f35181307e"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "20db777704827a4852bb273f66b728eb"
server_key_iv: "16136d1305e86143f351f0e7dc3b966f"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.289649 16164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:08.306399 16164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:08.310981 16164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:08.311296 16164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:08.312362 16164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:08.313783 16164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:08.313898 16164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.314070 16164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:08.314155 16164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.353993 16164 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:39107
I20260929 15:59:08.354030 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:39107 every 8 connection(s)
I20260929 15:59:08.355320 16164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 15:59:08.359047 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 16164
I20260929 15:59:08.359261 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 15:59:08.359871 5588 external_mini_cluster.cc:1428] Setting key 0af15d5d2ea8506278910d154c9d02c1
Sep 29 15:59:08 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697548, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:59:08.382906 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:08.358501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:52533 (local address 127.5.117.62:41295)
0929 15:59:08.359246 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:08.359253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:08.360728 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 15:59:08.363203 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:08.363208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:08.363281 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:08.363402 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:08.365978 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.366845 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.369589 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.370395 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.370536 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:08.375538 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:08.375544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:08.375548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:08.375595 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:08.379086 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:08.380195 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:08.380199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:08.380201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:08.380338 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:08.380856 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:08.380861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:08.380869 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:08.381170 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:08.381353 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:08.382265 (+ 912us) server_negotiation.cc:299] Negotiation successful
0929 15:59:08.382501 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":361,"threads_started":1}
I20260929 15:59:08.384824 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:08.359252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41295 (local address 127.5.117.2:52533)
0929 15:59:08.360477 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:08.360539 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:08.362761 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:08.363724 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:08.363739 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:08.364662 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:08.365679 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:08.365706 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.367063 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.367071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:08.369389 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:08.369406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.370690 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.370699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:08.371073 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:08.371988 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:08.372028 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:08.375216 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:08.379338 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:08.379351 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:08.379379 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:08.379967 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:08.380469 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:08.380474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:08.380477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:08.380710 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:08.381327 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:08.381337 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:08.381770 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 15:59:08.382109 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 15:59:08.382747 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":429,"threads_started":1}
I20260929 15:59:08.387265 16292 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41295
I20260929 15:59:08.387712 16292 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:08.388971 16292 heartbeater.cc:511] Master 127.5.117.62:41295 requested a full tablet report, sending...
I20260929 15:59:08.391290 15968 ts_manager.cc:194] Registered new tserver with Master: 45ed9b933b144bed8c8ae0f35181307e (127.5.117.2:39107)
I20260929 15:59:08.392381 15968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:52533
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 15:59:08.435738 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:59:08.800951 16158 heartbeater.cc:503] Master 127.5.117.62:41295 was elected leader, sending a full tablet report...
W20260929 15:59:08.821521 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:08.821843 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:08.822014 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:08.829259 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:08.829417 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:08.829586 16298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 15:59:08.839170 16298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:41295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:08.840816 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:08.842394 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:08.857189 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:08.857347 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:08.857502 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:08.859069 16298 server_base.cc:1034] running on GCE node
I20260929 15:59:08.860018 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:08.861541 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:08.862850 16298 hybrid_clock.cc:648] HybridClock initialized: now 1790697548862808 us; error 82 us; skew 500 ppm
I20260929 15:59:08.863360 16298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:08 dist-test-slave-b69v krb5kdc[15923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697548, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:08.867388 16298 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 15:59:08.869290 16298 webserver.cc:460] Webserver started at http://127.5.117.3:44909/ using document root <none> and password file <none>
I20260929 15:59:08.870291 16298 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:08.870486 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:08.870872 16298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:59:08.874343 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "cf579984b56a4bcc83626570d8d73390"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "4a044e6336759cfb49ba118f7500588f"
server_key_iv: "639d358617e439957b09465c5ff06f65"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.875192 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "cf579984b56a4bcc83626570d8d73390"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "4a044e6336759cfb49ba118f7500588f"
server_key_iv: "639d358617e439957b09465c5ff06f65"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.880790 16298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:59:08.884703 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.886474 16298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 15:59:08.886768 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "cf579984b56a4bcc83626570d8d73390"
format_stamp: "Formatted at 2026-09-29 15:59:08 on dist-test-slave-b69v"
server_key: "4a044e6336759cfb49ba118f7500588f"
server_key_iv: "639d358617e439957b09465c5ff06f65"
server_key_version: "encryptionkey@0"
I20260929 15:59:08.887064 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:08.911075 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:08.916574 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:08.916951 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:08.918237 16298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:08.919948 16298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:08.920094 16298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.920271 16298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:08.920356 16298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:08.961300 16298 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:40789
I20260929 15:59:08.961406 16425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:40789 every 8 connection(s)
I20260929 15:59:08.962749 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 15:59:08.965876 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 16298
I20260929 15:59:08.966106 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 15:59:08.966893 5588 external_mini_cluster.cc:1428] Setting key 602e64491c5fb6d163903ba55f2a72a5
Sep 29 15:59:08 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697548, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:59:08.986104 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:08.966169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:52247 (local address 127.5.117.62:41295)
0929 15:59:08.967516 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:08.967523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:08.968107 (+ 584us) server_negotiation.cc:407] Connection header received
0929 15:59:08.970350 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:08.970355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:08.970465 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:08.970601 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:08.972221 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.973211 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.974827 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.975233 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.975431 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:08.979691 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:08.979697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:08.979700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:08.979730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:08.982912 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:08.983666 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:08.983672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:08.983680 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:08.983795 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:08.984329 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:08.984334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:08.984337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:08.984613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:08.984798 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:08.985467 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 15:59:08.985681 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":1003,"threads_started":1}
I20260929 15:59:08.988150 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:08.966696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:41295 (local address 127.5.117.3:52247)
0929 15:59:08.967924 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:08.967971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:08.970028 (+ 2057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:08.970852 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:08.970860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:08.971385 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:08.971984 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:08.972002 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.973397 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.973403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:08.974656 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:08.974669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:08.975511 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:08.975518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:08.975758 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:08.976564 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:08.976587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:08.979363 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:08.983064 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:08.983073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:08.983087 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:08.983519 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:08.983916 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:08.983921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:08.983925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:08.984201 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:08.984757 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:08.984765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:08.985094 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 15:59:08.985359 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 15:59:08.985859 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":677,"threads_started":1}
I20260929 15:59:08.989974 16426 heartbeater.cc:347] Connected to a master server at 127.5.117.62:41295
I20260929 15:59:08.990250 16426 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:08.990970 16426 heartbeater.cc:511] Master 127.5.117.62:41295 requested a full tablet report, sending...
I20260929 15:59:08.992774 15968 ts_manager.cc:194] Registered new tserver with Master: cf579984b56a4bcc83626570d8d73390 (127.5.117.3:40789)
I20260929 15:59:08.994244 15968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:52247
I20260929 15:59:08.995563 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:59:09.012606 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.001080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38836 (local address 127.5.117.62:41295)
0929 15:59:09.001237 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.001241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.001811 (+ 570us) server_negotiation.cc:407] Connection header received
0929 15:59:09.001947 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.001950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.002017 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.002098 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:09.003556 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.004341 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.005521 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.005899 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.005997 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.008126 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:09.008131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:09.008134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:09.008162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:09.010488 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.011035 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.011038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.011040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.011140 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.011429 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.011432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.011434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.011721 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:09.011893 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.012206 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.012370 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 15:59:09.016655 15968 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:59:09.023772 15968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:59:09.048377 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.037850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40873 (local address 127.0.0.1:46260)
0929 15:59:09.038986 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:09.039008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:09.039140 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:09.041033 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:09.041039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:09.041068 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:09.041568 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.041583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.043852 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.043858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.045848 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.045860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.047443 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.047451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.047675 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.047740 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 15:59:09.047824 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:59:09.047952 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":174,"thread_start_us":2502,"threads_started":1}
I20260929 15:59:09.049582 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.038647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.5.117.1:40873)
0929 15:59:09.040189 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.040201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.040241 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:59:09.040391 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.040404 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.040722 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.041799 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:09.042071 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.043714 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.046009 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.047233 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.048896 (+ 1663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.049059 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.049119 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.049237 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":1081,"threads_started":1}
I20260929 15:59:09.049669 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.037068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:40789 (local address 127.0.0.1:34556)
0929 15:59:09.037872 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:09.037932 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:09.038133 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:09.041859 (+ 3726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:09.041863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:09.041889 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:09.042301 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.042309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.044243 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.044248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.046934 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.046949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.048806 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.048812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.048984 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.049024 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 15:59:09.049094 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 15:59:09.049226 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":177,"thread_start_us":364,"threads_started":1}
I20260929 15:59:09.050027 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.038428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:39107 (local address 127.0.0.1:47960)
0929 15:59:09.040594 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:09.040607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:09.040749 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:09.041379 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:09.041383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:09.041407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:09.041930 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.041941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.044164 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.044169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.046934 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.046949 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.049080 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.049089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.049243 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.049269 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 15:59:09.049367 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:59:09.049491 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"mutex_wait_us":191,"thread_start_us":848,"threads_started":1}
I20260929 15:59:09.050418 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.039090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.5.117.2:39107)
0929 15:59:09.039767 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.039777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.040708 (+ 931us) server_negotiation.cc:407] Connection header received
0929 15:59:09.040928 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.040939 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.041171 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.041368 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:09.042129 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.044003 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.047124 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.048577 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.048705 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.048942 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.049434 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.049614 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":303,"threads_started":1}
I20260929 15:59:09.052042 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.037492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.5.117.3:40789)
0929 15:59:09.041357 (+ 3865us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.041366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.041396 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:59:09.041499 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.041505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.041679 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.041860 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:09.042480 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.044050 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.047129 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.048570 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.049902 (+ 1332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.050141 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.050211 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.050383 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3683,"thread_start_us":3512,"threads_started":1}
I20260929 15:59:09.054554 16227 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a35a75b0d48de9b0246acf215a35a (DEFAULT_TABLE table=test-table [id=c09d3efedb1e4dda84112d056afa678d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:09.054582 16093 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a35a75b0d48de9b0246acf215a35a (DEFAULT_TABLE table=test-table [id=c09d3efedb1e4dda84112d056afa678d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:09.056113 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a35a75b0d48de9b0246acf215a35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:09.055975 16361 tablet_service.cc:1467] Processing CreateTablet for tablet ec1a35a75b0d48de9b0246acf215a35a (DEFAULT_TABLE table=test-table [id=c09d3efedb1e4dda84112d056afa678d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:09.056563 16093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a35a75b0d48de9b0246acf215a35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:09.057509 16361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1a35a75b0d48de9b0246acf215a35a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:09.069630 16445 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Bootstrap starting.
I20260929 15:59:09.072804 16446 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Bootstrap starting.
I20260929 15:59:09.074654 16447 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Bootstrap starting.
I20260929 15:59:09.075608 16445 tablet_bootstrap.cc:654] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:09.076864 16446 tablet_bootstrap.cc:654] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:09.077244 16445 log.cc:824] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:09.078320 16446 log.cc:824] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:09.078922 16447 tablet_bootstrap.cc:654] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:09.080367 16447 log.cc:824] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:09.088783 16446 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: No bootstrap required, opened a new log
I20260929 15:59:09.089205 16446 ts_tablet_manager.cc:1397] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260929 15:59:09.100667 16447 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: No bootstrap required, opened a new log
I20260929 15:59:09.101073 16447 ts_tablet_manager.cc:1397] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.013s
I20260929 15:59:09.101737 16445 tablet_bootstrap.cc:492] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: No bootstrap required, opened a new log
I20260929 15:59:09.102023 16445 ts_tablet_manager.cc:1397] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.008s
I20260929 15:59:09.103295 16446 raft_consensus.cc:348] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.103792 16446 raft_consensus.cc:374] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:09.103901 16446 raft_consensus.cc:729] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71135f09d8b84db392659072830ae3e4, State: Initialized, Role: FOLLOWER
I20260929 15:59:09.104765 16446 consensus_queue.cc:260] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.109771 16446 ts_tablet_manager.cc:1428] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260929 15:59:09.112308 16447 raft_consensus.cc:348] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.112476 16445 raft_consensus.cc:348] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.112774 16447 raft_consensus.cc:374] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:09.112897 16447 raft_consensus.cc:729] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf579984b56a4bcc83626570d8d73390, State: Initialized, Role: FOLLOWER
I20260929 15:59:09.112941 16445 raft_consensus.cc:374] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:09.113081 16445 raft_consensus.cc:729] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ed9b933b144bed8c8ae0f35181307e, State: Initialized, Role: FOLLOWER
I20260929 15:59:09.113628 16445 consensus_queue.cc:260] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.113644 16447 consensus_queue.cc:260] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.115339 16292 heartbeater.cc:503] Master 127.5.117.62:41295 was elected leader, sending a full tablet report...
I20260929 15:59:09.115989 16445 ts_tablet_manager.cc:1428] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 15:59:09.117126 16426 heartbeater.cc:503] Master 127.5.117.62:41295 was elected leader, sending a full tablet report...
I20260929 15:59:09.117813 16447 ts_tablet_manager.cc:1428] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
W20260929 15:59:09.214795 16427 tablet.cc:2369] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:59:09.215600 16452 raft_consensus.cc:482] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:59:09.215960 16452 raft_consensus.cc:504] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.218029 16452 leader_election.cc:288] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf579984b56a4bcc83626570d8d73390 (127.5.117.3:40789), 71135f09d8b84db392659072830ae3e4 (127.5.117.1:40873)
I20260929 15:59:09.228111 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.218516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:56075 (local address 127.5.117.3:40789)
0929 15:59:09.218752 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.218760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.219701 (+ 941us) server_negotiation.cc:407] Connection header received
0929 15:59:09.219926 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.219932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.220036 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.220174 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:09.221047 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.222270 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.225777 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.226892 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.227069 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.227199 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.227610 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.227771 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:59:09.228233 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.218553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:36013 (local address 127.5.117.1:40873)
0929 15:59:09.218764 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.218770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.219700 (+ 930us) server_negotiation.cc:407] Connection header received
0929 15:59:09.219926 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.219932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.220032 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.220175 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:09.221093 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.222270 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.225785 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.226941 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.227092 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.227203 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.227765 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.227909 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:59:09.228366 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.218463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:40789 (local address 127.5.117.2:56075)
0929 15:59:09.219541 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:09.219567 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:09.219765 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:09.220194 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:09.220200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:09.220234 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:09.220784 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.220803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.222590 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.222601 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.225561 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.225586 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.227145 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.227152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.227393 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.227440 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 15:59:09.227563 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:59:09.227763 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":257,"spinlock_wait_cycles":49792,"thread_start_us":657,"threads_started":1}
I20260929 15:59:09.228760 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.218470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40873 (local address 127.5.117.2:36013)
0929 15:59:09.219569 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:09.219590 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:09.219758 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:09.220194 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:09.220200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:09.220242 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:09.220905 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.220918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.222590 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.222601 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.225580 (+ 2979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:09.225589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.227377 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.227385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:09.227588 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.227620 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:59:09.227712 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 15:59:09.227833 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":85,"spinlock_wait_cycles":2944,"thread_start_us":491,"threads_started":1}
I20260929 15:59:09.229517 16381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a35a75b0d48de9b0246acf215a35a" candidate_uuid: "45ed9b933b144bed8c8ae0f35181307e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf579984b56a4bcc83626570d8d73390" is_pre_election: true
I20260929 15:59:09.229729 16113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a35a75b0d48de9b0246acf215a35a" candidate_uuid: "45ed9b933b144bed8c8ae0f35181307e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71135f09d8b84db392659072830ae3e4" is_pre_election: true
I20260929 15:59:09.230161 16381 raft_consensus.cc:2454] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ed9b933b144bed8c8ae0f35181307e in term 0.
I20260929 15:59:09.230306 16113 raft_consensus.cc:2454] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45ed9b933b144bed8c8ae0f35181307e in term 0.
I20260929 15:59:09.231030 16183 leader_election.cc:302] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45ed9b933b144bed8c8ae0f35181307e, 71135f09d8b84db392659072830ae3e4, cf579984b56a4bcc83626570d8d73390; no voters:
I20260929 15:59:09.231520 16452 raft_consensus.cc:2781] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:59:09.231691 16452 raft_consensus.cc:482] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:59:09.231828 16452 raft_consensus.cc:3037] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:09.234565 16452 raft_consensus.cc:504] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.235190 16452 leader_election.cc:288] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [CANDIDATE]: Term 1 election: Requested vote from peers cf579984b56a4bcc83626570d8d73390 (127.5.117.3:40789), 71135f09d8b84db392659072830ae3e4 (127.5.117.1:40873)
I20260929 15:59:09.235642 16113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a35a75b0d48de9b0246acf215a35a" candidate_uuid: "45ed9b933b144bed8c8ae0f35181307e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71135f09d8b84db392659072830ae3e4"
I20260929 15:59:09.235674 16381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1a35a75b0d48de9b0246acf215a35a" candidate_uuid: "45ed9b933b144bed8c8ae0f35181307e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf579984b56a4bcc83626570d8d73390"
I20260929 15:59:09.235839 16113 raft_consensus.cc:3037] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:09.235857 16381 raft_consensus.cc:3037] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:09.238162 16381 raft_consensus.cc:2454] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ed9b933b144bed8c8ae0f35181307e in term 1.
I20260929 15:59:09.238219 16113 raft_consensus.cc:2454] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45ed9b933b144bed8c8ae0f35181307e in term 1.
I20260929 15:59:09.238564 16181 leader_election.cc:302] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ed9b933b144bed8c8ae0f35181307e, cf579984b56a4bcc83626570d8d73390; no voters:
I20260929 15:59:09.238875 16452 raft_consensus.cc:2781] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:09.239358 16452 raft_consensus.cc:686] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 1 LEADER]: Becoming Leader. State: Replica: 45ed9b933b144bed8c8ae0f35181307e, State: Running, Role: LEADER
I20260929 15:59:09.239872 16452 consensus_queue.cc:237] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } }
I20260929 15:59:09.245033 15967 catalog_manager.cc:5495] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e reported cstate change: term changed from 0 to 1, leader changed from <none> to 45ed9b933b144bed8c8ae0f35181307e (127.5.117.2). New cstate: current_term: 1 leader_uuid: "45ed9b933b144bed8c8ae0f35181307e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45ed9b933b144bed8c8ae0f35181307e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 } health_report { overall_health: UNKNOWN } } }
W20260929 15:59:09.265374 16159 tablet.cc:2369] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:59:09.288460 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.282569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.5.117.2:39107)
0929 15:59:09.282771 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.282776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.282793 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:59:09.282894 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.282898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.282993 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.283130 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:59:09.283466 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.284248 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.285687 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.286089 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.286186 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.286821 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:59:09.287533 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:59:09.287720 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.287899 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.288122 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:59:09.298991 16381 raft_consensus.cc:1261] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 1 FOLLOWER]: Refusing update from remote peer 45ed9b933b144bed8c8ae0f35181307e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:59:09.299014 16113 raft_consensus.cc:1261] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 1 FOLLOWER]: Refusing update from remote peer 45ed9b933b144bed8c8ae0f35181307e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:59:09.300210 16456 consensus_queue.cc:1035] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf579984b56a4bcc83626570d8d73390" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 40789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:59:09.300648 16452 consensus_queue.cc:1035] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71135f09d8b84db392659072830ae3e4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:59:09.313366 16459 mvcc.cc:205] Tried to move back new op lower bound from 7334697161915297792 to 7334697161690517504. Current Snapshot: MvccSnapshot[applied={T|T < 7334697161915297792}]
I20260929 15:59:09.332267 15968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:59:09.332667 15968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:59:09.337422 15968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cf5b753901349daa578e65909ca0cb1: Sending DeleteTablet for 3 replicas of tablet ec1a35a75b0d48de9b0246acf215a35a
I20260929 15:59:09.338768 16093 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a35a75b0d48de9b0246acf215a35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:09 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:46260
I20260929 15:59:09.338768 16227 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a35a75b0d48de9b0246acf215a35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:09 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:47960
I20260929 15:59:09.338846 16361 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1a35a75b0d48de9b0246acf215a35a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:09 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:34556
I20260929 15:59:09.339879 16469 tablet_replica.cc:330] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: stopping tablet replica
I20260929 15:59:09.340567 16469 raft_consensus.cc:2229] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:59:09.341671 16469 raft_consensus.cc:2258] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:09.342715 16470 tablet_replica.cc:330] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: stopping tablet replica
I20260929 15:59:09.342940 16468 tablet_replica.cc:330] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: stopping tablet replica
I20260929 15:59:09.343282 16470 raft_consensus.cc:2229] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:59:09.343377 16468 raft_consensus.cc:2229] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:59:09.343914 16468 raft_consensus.cc:2258] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:09.343914 16470 raft_consensus.cc:2258] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:09.346292 16468 ts_tablet_manager.cc:1905] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:09.346472 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 16030
I20260929 15:59:09.346875 16470 ts_tablet_manager.cc:1905] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:09.354103 16468 ts_tablet_manager.cc:1918] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:09.354281 16468 log.cc:1196] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/ec1a35a75b0d48de9b0246acf215a35a
I20260929 15:59:09.354874 16468 ts_tablet_manager.cc:1939] T ec1a35a75b0d48de9b0246acf215a35a P 71135f09d8b84db392659072830ae3e4: Deleting consensus metadata
I20260929 15:59:09.357012 15955 catalog_manager.cc:4841] TS 71135f09d8b84db392659072830ae3e4 (127.5.117.1:40873): tablet ec1a35a75b0d48de9b0246acf215a35a (table test-table [id=c09d3efedb1e4dda84112d056afa678d]) successfully deleted
I20260929 15:59:09.358834 16469 ts_tablet_manager.cc:1905] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:09.359496 16470 ts_tablet_manager.cc:1918] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:09.359725 16470 log.cc:1196] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/ec1a35a75b0d48de9b0246acf215a35a
I20260929 15:59:09.360349 16470 ts_tablet_manager.cc:1939] T ec1a35a75b0d48de9b0246acf215a35a P cf579984b56a4bcc83626570d8d73390: Deleting consensus metadata
I20260929 15:59:09.362048 15953 catalog_manager.cc:4841] TS cf579984b56a4bcc83626570d8d73390 (127.5.117.3:40789): tablet ec1a35a75b0d48de9b0246acf215a35a (table test-table [id=c09d3efedb1e4dda84112d056afa678d]) successfully deleted
I20260929 15:59:09.371732 16469 ts_tablet_manager.cc:1918] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:09.371944 16469 log.cc:1196] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/ec1a35a75b0d48de9b0246acf215a35a
I20260929 15:59:09.372579 16469 ts_tablet_manager.cc:1939] T ec1a35a75b0d48de9b0246acf215a35a P 45ed9b933b144bed8c8ae0f35181307e: Deleting consensus metadata
I20260929 15:59:09.375025 15953 catalog_manager.cc:4841] TS 45ed9b933b144bed8c8ae0f35181307e (127.5.117.2:39107): tablet ec1a35a75b0d48de9b0246acf215a35a (table test-table [id=c09d3efedb1e4dda84112d056afa678d]) successfully deleted
Sep 29 15:59:09 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 15:59:09.387091 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.347734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.5.117.1:40873)
0929 15:59:09.347951 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.347958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.357746 (+ 9788us) server_negotiation.cc:407] Connection header received
0929 15:59:09.357819 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.357823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.357932 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.358029 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:09.361121 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.362317 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.364292 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.364974 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.365086 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.378138 (+ 13052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:09.378168 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:09.378193 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:09.378251 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:09.382788 (+ 4537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.384200 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.384222 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.384225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.384601 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.385151 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.385161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.385164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.385673 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:09.386029 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.386361 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.386667 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:59:09.388195 16153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:09.490411 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 16030
I20260929 15:59:09.518334 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 16164
Sep 29 15:59:09 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 15:59:09.535524 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.519313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.5.117.2:39107)
0929 15:59:09.519533 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.519541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.519563 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:59:09.519798 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.519809 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.519881 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.520043 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:09.521311 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.522224 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.523549 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.524006 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.524271 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.529138 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:09.529150 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:09.529168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:09.529214 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:09.532592 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.533282 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.533300 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.533303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.533477 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.533878 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.533883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.533886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.534298 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:09.534714 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.534888 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.535119 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 15:59:09.536362 16287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:09.662617 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 16164
I20260929 15:59:09.684973 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 16298
Sep 29 15:59:09 dist-test-slave-b69v krb5kdc[15923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 15:59:09.702306 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:09.685901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.5.117.3:40789)
0929 15:59:09.686118 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:09.686126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:09.686169 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:59:09.686403 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:09.686409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:09.686503 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:09.686637 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:09.688107 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.688872 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.690179 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:09.690611 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:09.690758 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:09.695627 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:09.695642 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:09.695664 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:09.695702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:09.698803 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.699655 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.699664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.699666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.699821 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:09.700480 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:09.700488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:09.700492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:09.700952 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:09.701304 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:09.701645 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 15:59:09.701907 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:59:09.703080 16421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:09.797866 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 16298
I20260929 15:59:09.818253 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 15936
I20260929 15:59:09.821672 15998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:09.939991 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 15936
2026-09-29T15:59:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4779 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:59:10 dist-test-slave-b69v krb5kdc[16478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:59:10 dist-test-slave-b69v krb5kdc[16478](info): set up 2 sockets
Sep 29 15:59:10 dist-test-slave-b69v krb5kdc[16478](info): commencing operation
krb5kdc: starting...
W20260929 15:59:11.089557 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.007s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 15:59:11 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:59:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangeradmin_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangeradmin_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangerlookup_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangerlookup_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
I20260929 15:59:11.524758 5588 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0NOJVQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/postgres -l logfile start
2026-09-29 15:59:14.273 UTC [16508] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:59:14.273 UTC [16508] LOG: listening on IPv4 address "127.5.117.20", port 59719
2026-09-29 15:59:14.277 UTC [16508] LOG: listening on Unix socket "/tmp/.s.PGSQL.59719"
2026-09-29 15:59:14.306 UTC [16510] LOG: database system was shut down at 2026-09-29 15:59:12 UTC
2026-09-29 15:59:14.310 UTC [16508] LOG: database system is ready to accept connections
I20260929 15:59:15.293286 5588 mini_postgres.cc:96] Postgres bound to 59719
2026-09-29 15:59:15.318 UTC [16519] FATAL: database "slave" does not exist
127.5.117.20:59719 - accepting connections
I20260929 15:59:15.319250 5588 mini_ranger.cc:161] Starting Ranger...
I20260929 15:59:15.353026 5588 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:59:15.897863 5588 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:59:15.898089 5588 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin
2026-09-29 15:59:16,251 [I] DB FLAVOR :POSTGRES
2026-09-29 15:59:16,252 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:59:16,252 [I] Checking connection..
2026-09-29 15:59:16,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:59:16,534 [I] Checking connection passed.
2026-09-29 15:59:16,534 [I] --------- Verifying version history table ---------
2026-09-29 15:59:16,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:16,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:59:16,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:17,091 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:59:17,091 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:59:17,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:59:17.358 UTC [16583] WARNING: there is no transaction in progress
2026-09-29 15:59:17,367 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:59:17,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:17,620 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:59:17,621 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:59:17,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:59:17,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:59:18,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b69v.c.gcp-upstream.internal','N') ;"
2026-09-29 15:59:18,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:18,748 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:59:18,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:19,016 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:59:19,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:19,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:19,547 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:59:19,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:59:20.597 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.615 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.633 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.650 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.673 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.695 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.759 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.769 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.786 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.798 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.810 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.827 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.838 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.844 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.850 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:20.857 UTC [16691] WARNING: there is no transaction in progress
2026-09-29 15:59:21,530 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:59:21,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:59:21,774 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:59:21,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:22,035 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:59:22,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:22,320 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:59:22,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:22,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:22,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b69v.c.gcp-upstream.internal';"
2026-09-29 15:59:23,106 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:59:23,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:59:23,382 [I] DB_PATCHES have already been applied
I20260929 15:59:23.389824 5588 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0NOJVQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:59:23.389991 5588 mini_ranger.cc:193] Using host: 127.5.117.20
I20260929 15:59:23.413975 5588 mini_ranger.cc:227] Ranger admin URL: http://127.5.117.20:44727
Sep 29, 2026 3:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:59:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:59:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:59:24.588279 5588 mini_ranger.cc:160] Time spent starting Ranger: real 9.269s user 0.006s sys 0.061s
Sep 29, 2026 3:59:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:59:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 3:59:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:59:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:59:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:59:48.793296 5588 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangerkms_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/rangerkms_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.117.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.117.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:59:48 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697588, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:59:49.008356 5588 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:59:49.035027 5588 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:59:49.885414 5588 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:59:49.885601 5588 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms
I20260929 15:59:49.885803 5588 mini_ranger_kms.cc:227] Using postgres at 127.5.117.20:59719
2026-09-29 15:59:50,411 [I] DB FLAVOR :POSTGRES
2026-09-29 15:59:50,411 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:59:50,411 [I] Checking connection
2026-09-29 15:59:50,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:59:50,690 [I] connection success
2026-09-29 15:59:50,691 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:59:50,691 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:59:50,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:59:50,966 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:59:50,966 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:59:50,966 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:59:50,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:59719/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:59:51,253 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:59:51.763229 5588 mini_ranger_kms.cc:310] Created kms service
I20260929 15:59:51.960198 5588 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:59:52.023280 5588 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:59:52 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697588, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 15:59:52.494858 5588 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:59:52.495229 5588 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms:/tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:59:52.495296 5588 mini_ranger_kms.cc:240] Using host: 127.5.117.20
I20260929 15:59:52.521623 5588 mini_ranger_kms.cc:276] Ranger KMS PID: 16872
I20260929 15:59:52.521894 5588 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.117.20:37085
Sep 29, 2026 3:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:59:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:59:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:59:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 15:59:54 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697594, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:54.897437 5588 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.889s user 0.142s sys 0.423s
I20260929 15:59:54.897753 5588 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:59:54.898013 5588 mini_ranger_kms.cc:401] 127.5.117.20:37085/kms/v1/keys
Sep 29 15:59:55 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697594, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
Sep 29 15:59:55 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697594, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
Sep 29 15:59:58 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 15:59:58.213441 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39585
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39585
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:37085/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:59:58.645977 16922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:58.646430 16922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:58.646600 16922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:58.655071 16922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:59:58.655221 16922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:58.655318 16922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:58.655411 16922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:59:58.655488 16922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:59:58.668361 16922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.117.20:37085/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39585
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 15:59:58.670275 16922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:58.672101 16922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:58.686676 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:58.686586 16927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:58.686641 16928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:58.688720 16922 server_base.cc:1034] running on GCE node
I20260929 15:59:58.689760 16922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:58.691546 16922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:58.692767 16922 hybrid_clock.cc:648] HybridClock initialized: now 1790697598692713 us; error 49 us; skew 500 ppm
I20260929 15:59:58.693188 16922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:58 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:58.696823 16922 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 15:59:58.698518 16922 webserver.cc:460] Webserver started at http://127.5.117.62:35875/ using document root <none> and password file <none>
I20260929 15:59:58.699280 16922 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:58.699429 16922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:58.699774 16922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:59:58 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 15:59:58.767936 16922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2"
format_stamp: "Formatted at 2026-09-29 15:59:58 on dist-test-slave-b69v"
server_key: "29bbe648ab2e630691625827fa300e4d"
server_key_iv: "522137eb2bcc68be54fb96b62706aad9"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:58.769306 16922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2"
format_stamp: "Formatted at 2026-09-29 15:59:58 on dist-test-slave-b69v"
server_key: "29bbe648ab2e630691625827fa300e4d"
server_key_iv: "522137eb2bcc68be54fb96b62706aad9"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:58.779376 16922 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260929 15:59:58.800596 16937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:58.802625 16922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:59:58.802875 16922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2"
format_stamp: "Formatted at 2026-09-29 15:59:58 on dist-test-slave-b69v"
server_key: "29bbe648ab2e630691625827fa300e4d"
server_key_iv: "522137eb2bcc68be54fb96b62706aad9"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:58.803171 16922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:58.820425 16922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:58.826110 16922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:58.826498 16922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:58.850602 16922 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39585
I20260929 15:59:58.850627 16988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39585 every 8 connection(s)
I20260929 15:59:58.852408 16922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 15:59:58.853509 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 16922
I20260929 15:59:58.853780 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 15:59:58.854568 5588 external_mini_cluster.cc:1428] Setting key 0391cc628104492cbb48720dd01a2467
I20260929 15:59:58.857648 16989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:59:58 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 15:59:58.872740 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: Bootstrap starting.
I20260929 15:59:58.876674 16989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:58.877884 16989 log.cc:824] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:58.880741 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:58.856916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.5.117.62:39585)
0929 15:59:58.857812 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:58.857825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:58.857871 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:59:58.859689 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:58.859712 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:58.860207 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:58.860578 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:58.862484 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:58.864301 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:58.866382 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:58.867255 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:58.867478 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:58.872619 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:58.872629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:58.872645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:58.872688 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:58.875956 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:58.876722 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:58.876730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:58.876757 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:58.876950 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:58.877348 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:58.877353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:58.877357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:58.877931 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:58.878233 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:58.878908 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 15:59:58.879475 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":401,"threads_started":1}
I20260929 15:59:58.881667 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: No bootstrap required, opened a new log
I20260929 15:59:58.890453 16989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } }
I20260929 15:59:58.890919 16989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:58.891040 16989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376ce2129dc44ae7b30c8ed57ed9c0b2, State: Initialized, Role: FOLLOWER
I20260929 15:59:58.891660 16989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } }
I20260929 15:59:58.891940 16989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:59:58.892171 16989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:59:58.892377 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:58.895267 16989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } }
I20260929 15:59:58.895834 16989 leader_election.cc:302] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376ce2129dc44ae7b30c8ed57ed9c0b2; no voters:
I20260929 15:59:58.896545 16989 leader_election.cc:288] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:59:58.896705 16994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:58.897533 16994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [term 1 LEADER]: Becoming Leader. State: Replica: 376ce2129dc44ae7b30c8ed57ed9c0b2, State: Running, Role: LEADER
I20260929 15:59:58.898151 16994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } }
I20260929 15:59:58.898986 16989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:59:58.901531 16996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376ce2129dc44ae7b30c8ed57ed9c0b2. Latest consensus state: current_term: 1 leader_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } } }
I20260929 15:59:58.901917 16996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:58.902752 16995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376ce2129dc44ae7b30c8ed57ed9c0b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39585 } } }
I20260929 15:59:58.903081 16995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2 [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:58.903842 16999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:59:58.911235 16999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:59:58.912037 16989 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:59:58.944136 16999 catalog_manager.cc:1347] Generated new cluster ID: eb2155576a2d4352967451bcfcb4bde6
I20260929 15:59:58.944377 16999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:59:58.966126 16999 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:59:58.968308 16999 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:59:58.978627 16999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: Generated new TSK 0
I20260929 15:59:58.980077 16999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 15:59:59.770638 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:37085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:00:00.259863 17034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:00.260342 17034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:00.260514 17034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:00.269471 17034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:00.269713 17034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:00.269928 17034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:00:00.280766 17034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.117.20:37085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:00.282305 17034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:00.283986 17034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:00.309566 17045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:00.310420 17044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:00.312254 17048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:00.314291 17034 server_base.cc:1034] running on GCE node
I20260929 16:00:00.315428 17034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:00.317118 17034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:00.320657 17034 hybrid_clock.cc:648] HybridClock initialized: now 1790697600320051 us; error 579 us; skew 500 ppm
I20260929 16:00:00.321213 17034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:00.326566 17034 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:00:00.333746 17034 webserver.cc:460] Webserver started at http://127.5.117.1:43749/ using document root <none> and password file <none>
I20260929 16:00:00.335139 17034 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:00.335322 17034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:00.335830 17034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:00.360014 17034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "e082556882464243ac74537a714e407f"
format_stamp: "Formatted at 2026-09-29 16:00:00 on dist-test-slave-b69v"
server_key: "9c924cb5c0f32889c657636391c85df7"
server_key_iv: "2f27f9684585f18e190242a2c5bd2434"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:00.361476 17034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "e082556882464243ac74537a714e407f"
format_stamp: "Formatted at 2026-09-29 16:00:00 on dist-test-slave-b69v"
server_key: "9c924cb5c0f32889c657636391c85df7"
server_key_iv: "2f27f9684585f18e190242a2c5bd2434"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:00.369035 17034 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260929 16:00:00.389021 17054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:00.391397 17034 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:00:00.391665 17034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "e082556882464243ac74537a714e407f"
format_stamp: "Formatted at 2026-09-29 16:00:00 on dist-test-slave-b69v"
server_key: "9c924cb5c0f32889c657636391c85df7"
server_key_iv: "2f27f9684585f18e190242a2c5bd2434"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:00.391963 17034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:00.431027 17034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:00.439916 17034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:00.440412 17034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:00.441987 17034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:00.444375 17034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:00.444479 17034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:00.444685 17034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:00.444757 17034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:00.549582 17034 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:45063
I20260929 16:00:00.549595 17166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:45063 every 8 connection(s)
I20260929 16:00:00.551697 17034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:00.560184 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 17034
I20260929 16:00:00.560417 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:00:00.561317 5588 external_mini_cluster.cc:1428] Setting key b6b8669fead902a3ec7d4949bbe277dd
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:00.609978 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:00.559616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:53013 (local address 127.5.117.62:39585)
0929 16:00:00.582041 (+ 22425us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:00.582050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:00.586700 (+ 4650us) server_negotiation.cc:407] Connection header received
0929 16:00:00.586805 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:00.586811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:00.586934 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:00.587056 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:00.590980 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:00.592173 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:00.594708 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:00.595362 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:00.595458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:00.600710 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:00.600718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:00.600723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:00.600800 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:00.604911 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:00.606259 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:00.606265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:00.606269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:00.606500 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:00.607183 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:00.607189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:00.607193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:00.607647 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:00.607817 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:00.609217 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0929 16:00:00.609518 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22220,"thread_start_us":22441,"threads_started":1}
I20260929 16:00:00.613246 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:00.558774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39585 (local address 127.5.117.1:53013)
0929 16:00:00.583082 (+ 24308us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:00.583165 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:00.586107 (+ 2942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:00.588707 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:00.588723 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:00.589512 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:00.590703 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:00.590742 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:00.592414 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:00.592419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:00.594530 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:00.594549 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:00.595847 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:00.595852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:00.596173 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:00.597056 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:00.597097 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:00.600376 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:00.605261 (+ 4885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:00.605271 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:00.605298 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:00.606067 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:00.606681 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:00.606687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:00.606691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:00.607037 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:00.607968 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:00.607978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:00.608653 (+ 675us) client_negotiation.cc:769] Sending connection context
0929 16:00:00.610670 (+ 2017us) client_negotiation.cc:240] Negotiation successful
0929 16:00:00.611396 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23805,"thread_start_us":28311,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 16:00:00.615937 17167 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39585
I20260929 16:00:00.616377 17167 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:00.617429 17167 heartbeater.cc:511] Master 127.5.117.62:39585 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
I20260929 16:00:00.620563 16954 ts_manager.cc:194] Registered new tserver with Master: e082556882464243ac74537a714e407f (127.5.117.1:45063)
I20260929 16:00:00.622753 16954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:53013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:00:00.665300 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:37085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:00 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697600, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
W20260929 16:00:01.213088 17173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:01.213665 17173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:01.213928 17173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:01.229035 17173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:01.229372 17173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:01.229672 17173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:00:01.247152 17173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.117.20:37085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:01.249555 17173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:01.251669 17173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:01.284222 17181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:01.284409 17180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:01.290161 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:01.290377 17173 server_base.cc:1034] running on GCE node
I20260929 16:00:01.291308 17173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:01.292788 17173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:01.294099 17173 hybrid_clock.cc:648] HybridClock initialized: now 1790697601293995 us; error 97 us; skew 500 ppm
I20260929 16:00:01.294528 17173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:01 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:01.298499 17173 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:00:01.308656 17173 webserver.cc:460] Webserver started at http://127.5.117.2:33115/ using document root <none> and password file <none>
I20260929 16:00:01.309729 17173 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:01.309929 17173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:01.310364 17173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:01 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:01.334560 17173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "bc1e37387cf040cbbda5a0a8cba20861"
format_stamp: "Formatted at 2026-09-29 16:00:01 on dist-test-slave-b69v"
server_key: "0e8594f1dbd191d6eb5094197c4149d8"
server_key_iv: "3682182b2ce5f27620670caee2f2c32b"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:01.336198 17173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "bc1e37387cf040cbbda5a0a8cba20861"
format_stamp: "Formatted at 2026-09-29 16:00:01 on dist-test-slave-b69v"
server_key: "0e8594f1dbd191d6eb5094197c4149d8"
server_key_iv: "3682182b2ce5f27620670caee2f2c32b"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:01.350028 17173 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260929 16:00:01.376763 17195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:01.379558 17173 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260929 16:00:01.379863 17173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "bc1e37387cf040cbbda5a0a8cba20861"
format_stamp: "Formatted at 2026-09-29 16:00:01 on dist-test-slave-b69v"
server_key: "0e8594f1dbd191d6eb5094197c4149d8"
server_key_iv: "3682182b2ce5f27620670caee2f2c32b"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:01.380234 17173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:01.405076 17173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:01.413167 17173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:01.413614 17173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:01.415328 17173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:01.417654 17173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:01.417779 17173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:01.417927 17173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:01.417995 17173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:01.474382 17173 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:41033
I20260929 16:00:01.474498 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:41033 every 8 connection(s)
I20260929 16:00:01.476902 17173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:01.485668 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 17173
I20260929 16:00:01.485920 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:00:01.486573 5588 external_mini_cluster.cc:1428] Setting key 24afbedbf1fbbbfcc17abe33566b63f2
Sep 29 16:00:01 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697601, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:01.511888 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:01.481107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:40433 (local address 127.5.117.62:39585)
0929 16:00:01.481870 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:01.481878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:01.483327 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 16:00:01.486150 (+ 2823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:01.486155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:01.486288 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:01.486441 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:01.490241 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:01.491329 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:01.493807 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:01.494607 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:01.494866 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:01.500312 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:01.500328 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:01.500333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:01.500397 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:01.505203 (+ 4806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:01.506903 (+ 1700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:01.506913 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:01.506920 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:01.507146 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:01.508526 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:01.508535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:01.508541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:01.509300 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:01.509576 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:01.510874 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0929 16:00:01.511260 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":345,"threads_started":1}
I20260929 16:00:01.513401 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:01.481997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39585 (local address 127.5.117.2:40433)
0929 16:00:01.483063 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:01.483139 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:01.485540 (+ 2401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:01.486848 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:01.486865 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:01.487950 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:01.489689 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:01.489749 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:01.491680 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:01.491687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:01.493520 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:01.493547 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:01.495091 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:01.495114 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:01.495538 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:01.496826 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:01.496867 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:01.499664 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:01.505595 (+ 5931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:01.505612 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:01.505641 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:01.506658 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:01.507612 (+ 954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:01.507620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:01.507625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:01.508258 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:01.509582 (+ 1324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:01.509602 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:01.510295 (+ 693us) client_negotiation.cc:769] Sending connection context
0929 16:00:01.510685 (+ 390us) client_negotiation.cc:240] Negotiation successful
0929 16:00:01.511435 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":443,"threads_started":1}
I20260929 16:00:01.516711 17310 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39585
I20260929 16:00:01.517339 17310 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:01.518740 17310 heartbeater.cc:511] Master 127.5.117.62:39585 requested a full tablet report, sending...
I20260929 16:00:01.521890 16954 ts_manager.cc:194] Registered new tserver with Master: bc1e37387cf040cbbda5a0a8cba20861 (127.5.117.2:41033)
I20260929 16:00:01.523309 16954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:40433
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:00:01.569341 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:37085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:01.627605 17167 heartbeater.cc:503] Master 127.5.117.62:39585 was elected leader, sending a full tablet report...
W20260929 16:00:02.001507 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:02.001969 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:02.002256 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:02.014001 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:02.014238 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:02.014475 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:00:02.027300 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.117.20:37085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:02.029424 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:02.031031 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:02.048800 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:02.049077 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:02.049250 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:02.049991 17316 server_base.cc:1034] running on GCE node
I20260929 16:00:02.050966 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:02.052171 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:02.053433 17316 hybrid_clock.cc:648] HybridClock initialized: now 1790697602053369 us; error 48 us; skew 500 ppm
I20260929 16:00:02.053905 17316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697602, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:02.059430 17316 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:00:02.062021 17316 webserver.cc:460] Webserver started at http://127.5.117.3:46175/ using document root <none> and password file <none>
I20260929 16:00:02.062983 17316 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:02.063162 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:02.063596 17316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697602, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:02.084371 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "337f3bd958224482a9bcb1bc69b1a2e1"
format_stamp: "Formatted at 2026-09-29 16:00:02 on dist-test-slave-b69v"
server_key: "284390492f4dbc58f03ebc747b684a6c"
server_key_iv: "4097d9567cfcfa2e62815367f1441730"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:02.085564 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "337f3bd958224482a9bcb1bc69b1a2e1"
format_stamp: "Formatted at 2026-09-29 16:00:02 on dist-test-slave-b69v"
server_key: "284390492f4dbc58f03ebc747b684a6c"
server_key_iv: "4097d9567cfcfa2e62815367f1441730"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:02.093374 17316 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260929 16:00:02.112438 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:02.114414 17316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:00:02.114626 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "337f3bd958224482a9bcb1bc69b1a2e1"
format_stamp: "Formatted at 2026-09-29 16:00:02 on dist-test-slave-b69v"
server_key: "284390492f4dbc58f03ebc747b684a6c"
server_key_iv: "4097d9567cfcfa2e62815367f1441730"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:02.114904 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:02.130617 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:02.136474 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:02.136910 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:02.138087 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:02.140220 17316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:02.140362 17316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:02.140539 17316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:02.140617 17316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:02.185843 17316 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:45945
I20260929 16:00:02.185894 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:45945 every 8 connection(s)
I20260929 16:00:02.187538 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:02.195463 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 17316
I20260929 16:00:02.195696 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:00:02.196254 5588 external_mini_cluster.cc:1428] Setting key 0269ba6305679672da14965e51426046
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697602, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:02.222257 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.190885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:48085 (local address 127.5.117.62:39585)
0929 16:00:02.191613 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.191622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.192776 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 16:00:02.194839 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.194945 (+ 106us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.195158 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.195318 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:02.197471 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.198574 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.201105 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.201629 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.201749 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.214690 (+ 12941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:02.214695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:02.214699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:02.214753 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:02.218012 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.219298 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.219303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.219305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.219453 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.220035 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.220039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.220041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.220452 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:02.220631 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.221479 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.221807 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":339,"threads_started":1}
I20260929 16:00:02.223650 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.191401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39585 (local address 127.5.117.3:48085)
0929 16:00:02.192555 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.192623 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.194455 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.195678 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.195698 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.196478 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:02.197268 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.197289 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.198824 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.198833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.200896 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.200914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.201915 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.201920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.202172 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.211220 (+ 9048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:02.211271 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:02.214384 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:02.218229 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:02.218239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:02.218261 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:02.219109 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:02.219618 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:02.219622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:02.219625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:02.219878 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:02.220617 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:02.220626 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:02.221057 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.221377 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.222148 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":454,"threads_started":1}
I20260929 16:00:02.226095 17444 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39585
I20260929 16:00:02.226577 17444 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:02.227730 17444 heartbeater.cc:511] Master 127.5.117.62:39585 requested a full tablet report, sending...
I20260929 16:00:02.229921 16953 ts_manager.cc:194] Registered new tserver with Master: 337f3bd958224482a9bcb1bc69b1a2e1 (127.5.117.3:45945)
I20260929 16:00:02.231069 16953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:48085
I20260929 16:00:02.241504 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:02.262935 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.247055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.5.117.62:39585)
0929 16:00:02.247379 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.247390 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.247855 (+ 465us) server_negotiation.cc:407] Connection header received
0929 16:00:02.248135 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.248140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.248257 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.248375 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:02.250403 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.251215 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.253023 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.253670 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.253801 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.256852 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:02.256858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:02.256862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:02.256904 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:02.259995 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.260971 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.260975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.260977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.261126 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.261458 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.261461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.261463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.261798 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:02.262022 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.262349 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.262574 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 16:00:02.268805 16953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:00:02.278103 16953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:00:02.310500 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.296400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.5.117.2:41033)
0929 16:00:02.297052 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.297066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.298072 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 16:00:02.298565 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.298577 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.298967 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.299169 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:02.300537 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.302573 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.306848 (+ 4275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.308924 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.309057 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.309434 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.309836 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.310054 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":1065,"threads_started":1}
I20260929 16:00:02.311864 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.296119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:41033 (local address 127.0.0.1:37270)
0929 16:00:02.297888 (+ 1769us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.297940 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.298275 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.299583 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.299595 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.299631 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:02.300263 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.300281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.302868 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.302876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.306589 (+ 3713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.306615 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.309277 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.309288 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.309525 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.309588 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.309768 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.311381 (+ 1613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1449,"thread_start_us":1235,"threads_started":1}
I20260929 16:00:02.318053 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.297281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.5.117.1:45063)
0929 16:00:02.308996 (+ 11715us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.309007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.309535 (+ 528us) server_negotiation.cc:407] Connection header received
0929 16:00:02.309683 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.309690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.310013 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.310208 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:02.311320 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.312711 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.315126 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.316678 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.316852 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.317174 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.317467 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.317654 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11484,"thread_start_us":11233,"threads_started":1}
I20260929 16:00:02.318051 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.296847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45063 (local address 127.0.0.1:54794)
0929 16:00:02.309299 (+ 12452us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.309318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.309536 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.310369 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.310378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.310407 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:02.310889 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.310903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.312857 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.312863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.314933 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.314946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.316943 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.316949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.317258 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.317296 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.317411 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.317568 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12284,"mutex_wait_us":65,"thread_start_us":10609,"threads_started":1}
I20260929 16:00:02.317865 17245 tablet_service.cc:1467] Processing CreateTablet for tablet 4d77304c371c4ee99c50b540a76dca19 (DEFAULT_TABLE table=test-table [id=f4e1e648c3e2406ca456c633c721ac14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:02.320094 17245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d77304c371c4ee99c50b540a76dca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:02.322335 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.296122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:45945 (local address 127.0.0.1:40420)
0929 16:00:02.307136 (+ 11014us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.307166 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.307339 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.310413 (+ 3074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.310417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.310438 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:02.311247 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.311260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.314738 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.314748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.317631 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.317662 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.320942 (+ 3280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.320949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.321160 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.321201 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.321579 (+ 378us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.321752 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10812,"mutex_wait_us":487,"thread_start_us":12344,"threads_started":1}
I20260929 16:00:02.323227 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.296612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.5.117.3:45945)
0929 16:00:02.308409 (+ 11797us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.308425 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.308476 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:00:02.308618 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.308628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.309774 (+ 1146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.310087 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:02.311473 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.314444 (+ 2971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.318013 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.320333 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.320484 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.320758 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.322625 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.322833 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11583,"thread_start_us":11323,"threads_started":1}
I20260929 16:00:02.323516 17102 tablet_service.cc:1467] Processing CreateTablet for tablet 4d77304c371c4ee99c50b540a76dca19 (DEFAULT_TABLE table=test-table [id=f4e1e648c3e2406ca456c633c721ac14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:02.325448 17102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d77304c371c4ee99c50b540a76dca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:02.328128 17379 tablet_service.cc:1467] Processing CreateTablet for tablet 4d77304c371c4ee99c50b540a76dca19 (DEFAULT_TABLE table=test-table [id=f4e1e648c3e2406ca456c633c721ac14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:02.329864 17379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d77304c371c4ee99c50b540a76dca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:02.334936 17463 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Bootstrap starting.
I20260929 16:00:02.339917 17463 tablet_bootstrap.cc:654] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:02.341612 17463 log.cc:824] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:02.343297 17464 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Bootstrap starting.
I20260929 16:00:02.346412 17466 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Bootstrap starting.
I20260929 16:00:02.347854 17464 tablet_bootstrap.cc:654] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:02.348074 17463 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: No bootstrap required, opened a new log
I20260929 16:00:02.348448 17463 ts_tablet_manager.cc:1397] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260929 16:00:02.349424 17464 log.cc:824] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:02.349988 17466 tablet_bootstrap.cc:654] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:02.351320 17466 log.cc:824] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:02.354480 17466 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: No bootstrap required, opened a new log
I20260929 16:00:02.354875 17466 ts_tablet_manager.cc:1397] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:00:02.357322 17464 tablet_bootstrap.cc:492] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: No bootstrap required, opened a new log
I20260929 16:00:02.357637 17464 ts_tablet_manager.cc:1397] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260929 16:00:02.361370 17463 raft_consensus.cc:348] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.361917 17463 raft_consensus.cc:374] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:02.362082 17463 raft_consensus.cc:729] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1e37387cf040cbbda5a0a8cba20861, State: Initialized, Role: FOLLOWER
I20260929 16:00:02.362838 17466 raft_consensus.cc:348] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.362848 17463 consensus_queue.cc:260] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.363195 17466 raft_consensus.cc:374] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:02.363323 17466 raft_consensus.cc:729] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337f3bd958224482a9bcb1bc69b1a2e1, State: Initialized, Role: FOLLOWER
I20260929 16:00:02.363952 17466 consensus_queue.cc:260] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.366370 17310 heartbeater.cc:503] Master 127.5.117.62:39585 was elected leader, sending a full tablet report...
I20260929 16:00:02.366920 17463 ts_tablet_manager.cc:1428] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260929 16:00:02.368165 17444 heartbeater.cc:503] Master 127.5.117.62:39585 was elected leader, sending a full tablet report...
I20260929 16:00:02.368857 17466 ts_tablet_manager.cc:1428] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260929 16:00:02.369612 17464 raft_consensus.cc:348] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.370143 17464 raft_consensus.cc:374] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:02.370326 17464 raft_consensus.cc:729] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e082556882464243ac74537a714e407f, State: Initialized, Role: FOLLOWER
I20260929 16:00:02.371003 17464 consensus_queue.cc:260] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.374047 17464 ts_tablet_manager.cc:1428] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260929 16:00:02.440037 17445 tablet.cc:2369] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:00:02.479969 17311 tablet.cc:2369] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:02.540585 17470 raft_consensus.cc:482] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:02.540953 17470 raft_consensus.cc:504] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.542862 17470 leader_election.cc:288] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc1e37387cf040cbbda5a0a8cba20861 (127.5.117.2:41033), e082556882464243ac74537a714e407f (127.5.117.1:45063)
I20260929 16:00:02.550472 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.543365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:41033 (local address 127.5.117.3:42755)
0929 16:00:02.543817 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.543841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.544027 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.544528 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.544533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.544565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:02.544835 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.544847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.546496 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.546502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.548312 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.548324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.549658 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.549664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.549870 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.549905 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.550004 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.550156 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":58}
I20260929 16:00:02.550523 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.543512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:42755 (local address 127.5.117.2:41033)
0929 16:00:02.543812 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.543818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.543939 (+ 121us) server_negotiation.cc:407] Connection header received
0929 16:00:02.544251 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.544259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.544363 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.544473 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:02.544993 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.546352 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.548478 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.549497 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.549647 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.549755 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.550053 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.550219 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 16:00:02.551954 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.543538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:42737 (local address 127.5.117.1:45063)
0929 16:00:02.543803 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.543811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.544621 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:00:02.544849 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.544855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.544952 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.545087 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:02.545697 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.546679 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.549273 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.550658 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.550813 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.550940 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.551508 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.551658 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:00:02.551980 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.543365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:45063 (local address 127.5.117.3:42737)
0929 16:00:02.544504 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:02.544521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:02.544666 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:02.545091 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:02.545095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:02.545116 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:02.545531 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.545540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.546839 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.546845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.549098 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:02.549112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.551138 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.551144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:02.551328 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.551363 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:00:02.551454 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:00:02.551572 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":107,"thread_start_us":658,"threads_started":1}
I20260929 16:00:02.551867 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d77304c371c4ee99c50b540a76dca19" candidate_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1e37387cf040cbbda5a0a8cba20861" is_pre_election: true
I20260929 16:00:02.552639 17265 raft_consensus.cc:2454] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 337f3bd958224482a9bcb1bc69b1a2e1 in term 0.
I20260929 16:00:02.553215 17122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d77304c371c4ee99c50b540a76dca19" candidate_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e082556882464243ac74537a714e407f" is_pre_election: true
I20260929 16:00:02.553475 17334 leader_election.cc:302] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337f3bd958224482a9bcb1bc69b1a2e1, bc1e37387cf040cbbda5a0a8cba20861; no voters:
I20260929 16:00:02.553759 17122 raft_consensus.cc:2454] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 337f3bd958224482a9bcb1bc69b1a2e1 in term 0.
I20260929 16:00:02.553918 17470 raft_consensus.cc:2781] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:00:02.554204 17470 raft_consensus.cc:482] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:00:02.554350 17470 raft_consensus.cc:3037] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:00:02.555689 17168 tablet.cc:2369] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:02.557482 17470 raft_consensus.cc:504] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.558233 17470 leader_election.cc:288] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [CANDIDATE]: Term 1 election: Requested vote from peers bc1e37387cf040cbbda5a0a8cba20861 (127.5.117.2:41033), e082556882464243ac74537a714e407f (127.5.117.1:45063)
I20260929 16:00:02.558964 17265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d77304c371c4ee99c50b540a76dca19" candidate_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc1e37387cf040cbbda5a0a8cba20861"
I20260929 16:00:02.558964 17122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d77304c371c4ee99c50b540a76dca19" candidate_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e082556882464243ac74537a714e407f"
I20260929 16:00:02.559337 17122 raft_consensus.cc:3037] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:02.559337 17265 raft_consensus.cc:3037] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:02.563514 17122 raft_consensus.cc:2454] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 337f3bd958224482a9bcb1bc69b1a2e1 in term 1.
I20260929 16:00:02.563514 17265 raft_consensus.cc:2454] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 337f3bd958224482a9bcb1bc69b1a2e1 in term 1.
I20260929 16:00:02.564215 17334 leader_election.cc:302] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337f3bd958224482a9bcb1bc69b1a2e1, bc1e37387cf040cbbda5a0a8cba20861; no voters:
I20260929 16:00:02.564606 17470 raft_consensus.cc:2781] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:02.565232 17470 raft_consensus.cc:686] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 1 LEADER]: Becoming Leader. State: Replica: 337f3bd958224482a9bcb1bc69b1a2e1, State: Running, Role: LEADER
I20260929 16:00:02.566087 17470 consensus_queue.cc:237] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } }
I20260929 16:00:02.571544 16954 catalog_manager.cc:5495] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 337f3bd958224482a9bcb1bc69b1a2e1 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "337f3bd958224482a9bcb1bc69b1a2e1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 } health_report { overall_health: UNKNOWN } } }
I20260929 16:00:02.610157 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.603508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.5.117.3:45945)
0929 16:00:02.603767 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.603772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.603810 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:00:02.604230 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.604234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.604410 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.604580 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:00:02.605062 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.605889 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.607550 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.608063 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.608207 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.608753 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:00:02.609363 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:00:02.609527 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.609628 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.609722 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:00:02.620528 17265 raft_consensus.cc:1261] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 1 FOLLOWER]: Refusing update from remote peer 337f3bd958224482a9bcb1bc69b1a2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:02.620520 17122 raft_consensus.cc:1261] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 1 FOLLOWER]: Refusing update from remote peer 337f3bd958224482a9bcb1bc69b1a2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:02.621793 17473 consensus_queue.cc:1035] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc1e37387cf040cbbda5a0a8cba20861" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:02.622241 17470 consensus_queue.cc:1035] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e082556882464243ac74537a714e407f" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:02.652530 16954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:02.652906 16954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:02.657137 16954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 376ce2129dc44ae7b30c8ed57ed9c0b2: Sending DeleteTablet for 3 replicas of tablet 4d77304c371c4ee99c50b540a76dca19
I20260929 16:00:02.658325 17379 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d77304c371c4ee99c50b540a76dca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:40420
I20260929 16:00:02.658349 17102 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d77304c371c4ee99c50b540a76dca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54794
I20260929 16:00:02.658344 17245 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d77304c371c4ee99c50b540a76dca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:37270
I20260929 16:00:02.663220 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 17034
I20260929 16:00:02.671530 17486 tablet_replica.cc:330] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: stopping tablet replica
I20260929 16:00:02.672463 17487 tablet_replica.cc:330] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: stopping tablet replica
I20260929 16:00:02.673009 17487 raft_consensus.cc:2229] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:02.673612 17487 raft_consensus.cc:2258] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:02.674535 17488 tablet_replica.cc:330] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: stopping tablet replica
I20260929 16:00:02.674726 17486 raft_consensus.cc:2229] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:00:02.674971 17488 raft_consensus.cc:2229] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:02.675617 17488 raft_consensus.cc:2258] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:02.675961 17486 raft_consensus.cc:2258] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:02.677524 17487 ts_tablet_manager.cc:1905] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:02.678161 17486 ts_tablet_manager.cc:1905] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:00:02.685508 17487 ts_tablet_manager.cc:1918] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:02.685719 17487 log.cc:1196] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/4d77304c371c4ee99c50b540a76dca19
I20260929 16:00:02.686393 17487 ts_tablet_manager.cc:1939] T 4d77304c371c4ee99c50b540a76dca19 P e082556882464243ac74537a714e407f: Deleting consensus metadata
I20260929 16:00:02.686424 17486 ts_tablet_manager.cc:1918] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:02.686645 17486 log.cc:1196] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/4d77304c371c4ee99c50b540a76dca19
I20260929 16:00:02.687311 17486 ts_tablet_manager.cc:1939] T 4d77304c371c4ee99c50b540a76dca19 P 337f3bd958224482a9bcb1bc69b1a2e1: Deleting consensus metadata
I20260929 16:00:02.689994 16938 catalog_manager.cc:4841] TS 337f3bd958224482a9bcb1bc69b1a2e1 (127.5.117.3:45945): tablet 4d77304c371c4ee99c50b540a76dca19 (table test-table [id=f4e1e648c3e2406ca456c633c721ac14]) successfully deleted
I20260929 16:00:02.691097 16939 catalog_manager.cc:4841] TS e082556882464243ac74537a714e407f (127.5.117.1:45063): tablet 4d77304c371c4ee99c50b540a76dca19 (table test-table [id=f4e1e648c3e2406ca456c633c721ac14]) successfully deleted
I20260929 16:00:02.691138 17488 ts_tablet_manager.cc:1905] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:02.694168 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.666322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.5.117.1:45063)
0929 16:00:02.666561 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.666571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.671701 (+ 5130us) server_negotiation.cc:407] Connection header received
0929 16:00:02.671917 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.671926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.672069 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.672219 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:02.675027 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.676333 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.678342 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.679084 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.679197 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.685280 (+ 6083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:02.685294 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:02.685314 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:02.685355 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:02.690373 (+ 5018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.691637 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.691645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.691647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.691861 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.692334 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.692340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.692343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.692771 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:02.693112 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.693455 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.693708 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:00:02.695015 17162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:02.701386 17488 ts_tablet_manager.cc:1918] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:02.701581 17488 log.cc:1196] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/4d77304c371c4ee99c50b540a76dca19
I20260929 16:00:02.702171 17488 ts_tablet_manager.cc:1939] T 4d77304c371c4ee99c50b540a76dca19 P bc1e37387cf040cbbda5a0a8cba20861: Deleting consensus metadata
I20260929 16:00:02.704504 16940 catalog_manager.cc:4841] TS bc1e37387cf040cbbda5a0a8cba20861 (127.5.117.2:41033): tablet 4d77304c371c4ee99c50b540a76dca19 (table test-table [id=f4e1e648c3e2406ca456c633c721ac14]) successfully deleted
I20260929 16:00:02.797401 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 17034
I20260929 16:00:02.819701 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 17173
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:00:02.837714 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.820819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.5.117.2:41033)
0929 16:00:02.821057 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.821063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.821086 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:00:02.821279 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.821290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.821365 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.821495 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:02.822937 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.823720 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.825155 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.825622 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.825771 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.830837 (+ 5066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:02.830856 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:02.830878 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:02.830921 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:02.834435 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.835209 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.835217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.835220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.835361 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.835801 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.835807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.835810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.836320 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:02.836661 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.837023 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.837298 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:00:02.838677 17305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:02.955560 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 17173
I20260929 16:00:02.976828 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 17316
Sep 29 16:00:02 dist-test-slave-b69v krb5kdc[16478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:00:02.997980 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:02.978139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.5.117.3:45945)
0929 16:00:02.978394 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:02.978402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:02.978427 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:00:02.978599 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:02.978614 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:02.978689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:02.978815 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:02.980468 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.981418 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.983000 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:02.983509 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:02.983684 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:02.989846 (+ 6162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:02.989863 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:02.989889 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:02.989950 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:02.994205 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.995240 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.995254 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.995256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.995459 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:02.995910 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:02.995915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:02.995920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:02.996502 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:02.996902 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:02.997242 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:00:02.997528 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:00:02.999063 17439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:03.142043 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 17316
I20260929 16:00:03.165764 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 16922
I20260929 16:00:03.167155 16984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:03.294013 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 16922
I20260929 16:00:03.319698 5588 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:00:03.666320 5588 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:00:03.666621 5588 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:00:04.013309 5588 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:00:04.013 UTC [16508] LOG: received smart shutdown request
2026-09-29 16:00:04.018 UTC [16508] LOG: background worker "logical replication launcher" (PID 16517) exited with exit code 1
2026-09-29 16:00:04.018 UTC [16511] LOG: shutting down
2026-09-29 16:00:04.249 UTC [16508] LOG: database system is shut down
2026-09-29T16:00:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54453 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:00:04 dist-test-slave-b69v krb5kdc[17502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:00:04 dist-test-slave-b69v krb5kdc[17502](info): set up 2 sockets
Sep 29 16:00:04 dist-test-slave-b69v krb5kdc[17502](info): commencing operation
krb5kdc: starting...
W20260929 16:00:05.552990 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:00:05 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:00:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangeradmin_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangeradmin_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangerlookup_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangerlookup_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
I20260929 16:00:05.972280 5588 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0NOJVQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/postgres -l logfile start
2026-09-29 16:00:08.721 UTC [17532] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:00:08.722 UTC [17532] LOG: listening on IPv4 address "127.5.117.20", port 44269
2026-09-29 16:00:08.726 UTC [17532] LOG: listening on Unix socket "/tmp/.s.PGSQL.44269"
2026-09-29 16:00:08.754 UTC [17535] LOG: database system was shut down at 2026-09-29 16:00:07 UTC
2026-09-29 16:00:08.759 UTC [17532] LOG: database system is ready to accept connections
I20260929 16:00:09.740643 5588 mini_postgres.cc:96] Postgres bound to 44269
2026-09-29 16:00:09.765 UTC [17543] FATAL: database "slave" does not exist
127.5.117.20:44269 - accepting connections
I20260929 16:00:09.766469 5588 mini_ranger.cc:161] Starting Ranger...
I20260929 16:00:09.799197 5588 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:00:10.360287 5588 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:00:10.360468 5588 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin
2026-09-29 16:00:10,649 [I] DB FLAVOR :POSTGRES
2026-09-29 16:00:10,650 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:00:10,650 [I] Checking connection..
2026-09-29 16:00:10,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:00:10,918 [I] Checking connection passed.
2026-09-29 16:00:10,918 [I] --------- Verifying version history table ---------
2026-09-29 16:00:10,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:00:11,196 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:00:11,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:00:11,466 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:00:11,466 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:00:11,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:00:11.711 UTC [17607] WARNING: there is no transaction in progress
2026-09-29 16:00:11,719 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:00:11,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:00:11,992 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:00:11,992 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:00:11,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:00:12,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:00:12,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b69v.c.gcp-upstream.internal','N') ;"
2026-09-29 16:00:12,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:00:13,039 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:00:13,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:00:13,296 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:00:13,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:00:13,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:00:13,866 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:00:13,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:00:14.985 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.005 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.027 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.046 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.066 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.085 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.147 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.153 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.167 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.177 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.186 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.200 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.210 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.217 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.225 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15.232 UTC [17715] WARNING: there is no transaction in progress
2026-09-29 16:00:15,955 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:00:15,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:00:16,209 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:00:16,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:00:16,495 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:00:16,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:00:16,778 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:00:16,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:00:17,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:00:17,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b69v.c.gcp-upstream.internal';"
2026-09-29 16:00:17,563 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:00:17,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:00:17,808 [I] DB_PATCHES have already been applied
I20260929 16:00:17.815280 5588 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0NOJVQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:00:17.815392 5588 mini_ranger.cc:193] Using host: 127.5.117.20
I20260929 16:00:17.837538 5588 mini_ranger.cc:227] Ranger admin URL: http://127.5.117.20:36017
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:00:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:00:19.032203 5588 mini_ranger.cc:160] Time spent starting Ranger: real 9.266s user 0.005s sys 0.060s
Sep 29, 2026 4:00:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:00:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:00:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:00:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:00:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:00:43.030777 5588 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.117.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.117.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangerkms_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.117.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/rangerkms_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.117.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.117.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.117.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/HTTP_127.5.117.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:00:43 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697643, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:00:43.254662 5588 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:00:43.280383 5588 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:00:44.198057 5588 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:00:44.198233 5588 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms
I20260929 16:00:44.198288 5588 mini_ranger_kms.cc:227] Using postgres at 127.5.117.20:44269
2026-09-29 16:00:44,668 [I] DB FLAVOR :POSTGRES
2026-09-29 16:00:44,668 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:00:44,668 [I] Checking connection
2026-09-29 16:00:44,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:00:44,959 [I] connection success
2026-09-29 16:00:44,959 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:00:44,959 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:00:44,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:00:45,222 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:00:45,222 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:00:45,222 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:00:45,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.117.20:44269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:00:45,512 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:00:46.097059 5588 mini_ranger_kms.cc:310] Created kms service
I20260929 16:00:46.363436 5588 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:00:46.409462 5588 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:00:46 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697643, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:46.850920 5588 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:00:46.851308 5588 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms:/tmp/dist-test-task0NOJVQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0NOJVQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:00:46.851378 5588 mini_ranger_kms.cc:240] Using host: 127.5.117.20
I20260929 16:00:46.872529 5588 mini_ranger_kms.cc:276] Ranger KMS PID: 17896
I20260929 16:00:46.872711 5588 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.117.20:46069
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:00:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:00:49.201337 5588 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.947s user 0.098s sys 0.386s
I20260929 16:00:49.201668 5588 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:00:49.201879 5588 mini_ranger_kms.cc:401] 127.5.117.20:46069/kms/v1/keys
Sep 29 16:00:49 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:00:49 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
Sep 29 16:00:50 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.117.20@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
Sep 29 16:00:52 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:00:52.827952 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46145
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46145
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:53.229578 17946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:53.229895 17946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:53.230051 17946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:53.237380 17946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:53.237479 17946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:53.237530 17946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:53.237617 17946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:53.237699 17946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:53.247841 17946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.117.20:46069/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46145
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:53.249217 17946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:53.250670 17946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:53.264294 17951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:53.264343 17954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:53.264364 17952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:53.265218 17946 server_base.cc:1034] running on GCE node
I20260929 16:00:53.266083 17946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:53.267740 17946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:53.269021 17946 hybrid_clock.cc:648] HybridClock initialized: now 1790697653268984 us; error 46 us; skew 500 ppm
I20260929 16:00:53.269446 17946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:53 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:53.276533 17946 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:00:53.278163 17946 webserver.cc:460] Webserver started at http://127.5.117.62:40221/ using document root <none> and password file <none>
I20260929 16:00:53.278867 17946 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:53.278966 17946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:53.279264 17946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:53 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697653, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:53.334374 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "9043f2d24f924301bb5d2a1f5f0d41cb"
format_stamp: "Formatted at 2026-09-29 16:00:53 on dist-test-slave-b69v"
server_key: "b3d0d3cf6a21f89b81de2d58be7b3d9c"
server_key_iv: "b8eb4466e13c296e08bb5efac7f03959"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:53.335302 17946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "9043f2d24f924301bb5d2a1f5f0d41cb"
format_stamp: "Formatted at 2026-09-29 16:00:53 on dist-test-slave-b69v"
server_key: "b3d0d3cf6a21f89b81de2d58be7b3d9c"
server_key_iv: "b8eb4466e13c296e08bb5efac7f03959"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:53.342000 17946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:00:53.358549 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:53.360208 17946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:00:53.360409 17946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "9043f2d24f924301bb5d2a1f5f0d41cb"
format_stamp: "Formatted at 2026-09-29 16:00:53 on dist-test-slave-b69v"
server_key: "b3d0d3cf6a21f89b81de2d58be7b3d9c"
server_key_iv: "b8eb4466e13c296e08bb5efac7f03959"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:53.360653 17946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:53.377712 17946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:53.382135 17946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:53.382476 17946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:53.403236 17946 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46145
I20260929 16:00:53.403285 18012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46145 every 8 connection(s)
I20260929 16:00:53.404663 17946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:53.409606 18013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:53.411985 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 17946
I20260929 16:00:53.412463 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:00:53.413275 5588 external_mini_cluster.cc:1428] Setting key 99faf9e5400bd2b1abf40772945117b6
I20260929 16:00:53.424373 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Bootstrap starting.
I20260929 16:00:53.428375 18013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:53.429478 18013 log.cc:824] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:00:53 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:53.433163 18013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: No bootstrap required, opened a new log
I20260929 16:00:53.440608 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.415666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.5.117.62:46145)
0929 16:00:53.416690 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.416710 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.416963 (+ 253us) server_negotiation.cc:407] Connection header received
0929 16:00:53.419451 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.419484 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.420119 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.420628 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.421971 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.423958 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.425627 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.426767 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.426982 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.431655 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.431665 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.431681 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.431717 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.435582 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.436430 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.436437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.436443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.436607 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.437167 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.437174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.437180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.438019 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.438362 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.438866 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.439303 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":431,"threads_started":1}
I20260929 16:00:53.442847 18013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:53.443235 18013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:53.443353 18013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9043f2d24f924301bb5d2a1f5f0d41cb, State: Initialized, Role: FOLLOWER
I20260929 16:00:53.444020 18013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:53.444327 18013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:53.444494 18013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:53.444679 18013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:53.446913 18013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:53.447431 18013 leader_election.cc:302] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9043f2d24f924301bb5d2a1f5f0d41cb; no voters:
I20260929 16:00:53.448115 18013 leader_election.cc:288] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:00:53.448218 18018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:53.448977 18018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 LEADER]: Becoming Leader. State: Replica: 9043f2d24f924301bb5d2a1f5f0d41cb, State: Running, Role: LEADER
I20260929 16:00:53.449788 18018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:53.450520 18013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:53.452860 18019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } } }
I20260929 16:00:53.453292 18019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:53.453814 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9043f2d24f924301bb5d2a1f5f0d41cb. Latest consensus state: current_term: 1 leader_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } } }
I20260929 16:00:53.454156 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:53.455174 18025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:53.461449 18025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:53.462430 18013 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:00:53.481403 18025 catalog_manager.cc:1347] Generated new cluster ID: f8e2c406ec6544498e4cb465f5c0b39e
I20260929 16:00:53.482255 18025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:53.497206 18025 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:00:53.498610 18025 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:53.511646 18025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Generated new TSK 0
I20260929 16:00:53.513064 18025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:00:54.324685 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:54 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:00:54.803907 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:54.804503 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:54.804715 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:54.816485 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:54.816685 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:54.816951 18058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:00:54.833966 18058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:54.836287 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:54.838321 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:54.856720 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:54.858067 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:54.862691 18058 server_base.cc:1034] running on GCE node
W20260929 16:00:54.862968 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:54.863655 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:54.864871 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:54.866250 18058 hybrid_clock.cc:648] HybridClock initialized: now 1790697654866148 us; error 61 us; skew 500 ppm
I20260929 16:00:54.866983 18058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:54 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:54.871127 18058 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:00:54.873119 18058 webserver.cc:460] Webserver started at http://127.5.117.1:41279/ using document root <none> and password file <none>
I20260929 16:00:54.873977 18058 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:54.874126 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:54.874511 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:54 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:54.900754 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9"
format_stamp: "Formatted at 2026-09-29 16:00:54 on dist-test-slave-b69v"
server_key: "98a920ed89a6d6697f31b97f1f2cde86"
server_key_iv: "e8848e0d6853313d66d00467267101f9"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:54.902954 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9"
format_stamp: "Formatted at 2026-09-29 16:00:54 on dist-test-slave-b69v"
server_key: "98a920ed89a6d6697f31b97f1f2cde86"
server_key_iv: "e8848e0d6853313d66d00467267101f9"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:54.912546 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260929 16:00:54.944712 18078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:54.946861 18058 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.012s sys 0.002s
I20260929 16:00:54.947114 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9"
format_stamp: "Formatted at 2026-09-29 16:00:54 on dist-test-slave-b69v"
server_key: "98a920ed89a6d6697f31b97f1f2cde86"
server_key_iv: "e8848e0d6853313d66d00467267101f9"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:54.947458 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:54.987293 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:54.994067 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:54.994601 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:54.995935 18058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:54.997864 18058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:54.998011 18058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:00:54.998174 18058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:54.998266 18058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:55.162010 18058 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:33961
I20260929 16:00:55.162042 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:33961 every 8 connection(s)
I20260929 16:00:55.164176 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:55.168864 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18058
I20260929 16:00:55.169147 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:00:55.170092 5588 external_mini_cluster.cc:1428] Setting key b2830ac7a38cfc43551b93553506f4ac
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
I20260929 16:00:55.224336 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.174922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.1:56393)
0929 16:00:55.197329 (+ 22407us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.197404 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:55.200287 (+ 2883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:55.201617 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:55.201632 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:55.202378 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:55.203322 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:55.203356 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.205032 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.205038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:55.207051 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:55.207076 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.208422 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.208429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:55.208734 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:55.209698 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:55.209735 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:55.213192 (+ 3457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:55.217918 (+ 4726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:55.217932 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:55.217955 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:55.218689 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:55.219226 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:55.219233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:55.219239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:55.219527 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:55.220412 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:55.220422 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:55.220958 (+ 536us) client_negotiation.cc:769] Sending connection context
0929 16:00:55.221326 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 16:00:55.222014 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21987,"thread_start_us":21685,"threads_started":1}
I20260929 16:00:55.224493 18194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.175565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:56393 (local address 127.5.117.62:46145)
0929 16:00:55.196536 (+ 20971us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:55.196545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:55.200145 (+ 3600us) server_negotiation.cc:407] Connection header received
0929 16:00:55.200824 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:55.200830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:55.200953 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:55.201076 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:55.203544 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.204841 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.207260 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.207927 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.208069 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:55.213452 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:55.213461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:55.213469 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:55.213516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:55.217721 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:55.218867 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:55.218876 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:55.218881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:55.219067 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:55.219698 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:55.219704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:55.219708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:55.220168 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:55.220334 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:55.223847 (+ 3513us) server_negotiation.cc:299] Negotiation successful
0929 16:00:55.224094 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20798,"thread_start_us":20649,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
I20260929 16:00:55.226668 18191 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:00:55.227027 18191 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:55.227960 18191 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:00:55.230741 17978 ts_manager.cc:194] Registered new tserver with Master: 3cf9a4cb02a24b0ba6c5303ccd57eac9 (127.5.117.1:33961)
I20260929 16:00:55.232923 17978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:56393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:00:55.268455 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
W20260929 16:00:55.752033 18197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:55.752377 18197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:55.752550 18197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:55.760051 18197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:55.760233 18197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:55.760417 18197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:00:55.769764 18197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:55.771174 18197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:55.772751 18197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:55.788328 18212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.788365 18211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.789139 18214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:55.790500 18197 server_base.cc:1034] running on GCE node
I20260929 16:00:55.791198 18197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:55.792373 18197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:55.793638 18197 hybrid_clock.cc:648] HybridClock initialized: now 1790697655793584 us; error 45 us; skew 500 ppm
I20260929 16:00:55.794049 18197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:55.797976 18197 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:00:55.799816 18197 webserver.cc:460] Webserver started at http://127.5.117.2:45063/ using document root <none> and password file <none>
I20260929 16:00:55.800742 18197 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:55.800879 18197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:55.801271 18197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:55.822738 18197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "4112eb4246564248b819594c300d29ca"
format_stamp: "Formatted at 2026-09-29 16:00:55 on dist-test-slave-b69v"
server_key: "58a744ad5e86b6adcf37451764d78737"
server_key_iv: "a0395ea7db577caad0af09364477fd29"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:55.823658 18197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "4112eb4246564248b819594c300d29ca"
format_stamp: "Formatted at 2026-09-29 16:00:55 on dist-test-slave-b69v"
server_key: "58a744ad5e86b6adcf37451764d78737"
server_key_iv: "a0395ea7db577caad0af09364477fd29"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:55.829931 18197 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:00:55.853864 18221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.857676 18197 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s
I20260929 16:00:55.857919 18197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "4112eb4246564248b819594c300d29ca"
format_stamp: "Formatted at 2026-09-29 16:00:55 on dist-test-slave-b69v"
server_key: "58a744ad5e86b6adcf37451764d78737"
server_key_iv: "a0395ea7db577caad0af09364477fd29"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:55.858225 18197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:55.900645 18197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:55.905762 18197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:55.906163 18197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:55.907652 18197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:55.909616 18197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:55.909705 18197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.909875 18197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:55.909929 18197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.953459 18197 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44899
I20260929 16:00:55.953490 18333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44899 every 8 connection(s)
I20260929 16:00:55.955255 18197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:55.955585 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18197
I20260929 16:00:55.955812 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:00:55.956949 5588 external_mini_cluster.cc:1428] Setting key 728d6e8774ac9c87e51d6f3d4efdad1d
Sep 29 16:00:55 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:55.984313 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.959907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:34107 (local address 127.5.117.62:46145)
0929 16:00:55.964017 (+ 4110us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:55.964027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:55.965238 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 16:00:55.968481 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:55.968488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:55.968577 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:55.968713 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:55.970774 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.971582 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.973372 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.973820 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.973966 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:55.977662 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:55.977667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:55.977670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:55.977703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:55.980512 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:55.981309 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:55.981317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:55.981323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:55.981475 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:55.981976 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:55.981984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:55.981990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:55.982393 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:55.982551 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:55.983418 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 16:00:55.983735 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3911,"thread_start_us":3739,"threads_started":1}
I20260929 16:00:55.985560 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.959238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.2:34107)
0929 16:00:55.964943 (+ 5705us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.965029 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:55.968136 (+ 3107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:55.969009 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:55.969024 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:55.969728 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:55.970586 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:55.970609 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.971764 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.971770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:55.973209 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:55.973229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:55.974065 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:55.974075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:55.974333 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:55.975095 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:55.975119 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:55.977423 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:55.980666 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:55.980675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:55.980692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:55.981178 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:55.981600 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:55.981604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:55.981607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:55.981832 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:55.982549 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:55.982556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:55.982938 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 16:00:55.983236 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 16:00:55.983857 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5175,"thread_start_us":4892,"threads_started":1}
I20260929 16:00:55.987607 18334 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:00:55.987970 18334 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:55.988888 18334 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
I20260929 16:00:55.990784 17978 ts_manager.cc:194] Registered new tserver with Master: 4112eb4246564248b819594c300d29ca (127.5.117.2:44899)
I20260929 16:00:55.991747 17978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:34107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:00:56.027794 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:56.236919 18191 heartbeater.cc:503] Master 127.5.117.62:46145 was elected leader, sending a full tablet report...
W20260929 16:00:56.417672 18340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:56.418016 18340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:56.418190 18340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:56.425417 18340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:56.425591 18340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:56.425781 18340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:00:56.435257 18340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:56.436623 18340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:56.438038 18340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:56.452605 18345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:56.452566 18346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:56.452687 18348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:56.458536 18340 server_base.cc:1034] running on GCE node
I20260929 16:00:56.459619 18340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:56.461022 18340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:56.462317 18340 hybrid_clock.cc:648] HybridClock initialized: now 1790697656462267 us; error 69 us; skew 500 ppm
I20260929 16:00:56.462728 18340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:56.466640 18340 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:00:56.468374 18340 webserver.cc:460] Webserver started at http://127.5.117.3:43713/ using document root <none> and password file <none>
I20260929 16:00:56.469113 18340 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:56.469219 18340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:56.469516 18340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:56.489360 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "b5dc66531b4746aeb102e6dd5aca0153"
format_stamp: "Formatted at 2026-09-29 16:00:56 on dist-test-slave-b69v"
server_key: "5b438804cacb3dbf464c87df18f5f511"
server_key_iv: "94ffe4e7c5d70e92d3cbb28e13d82b32"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:56.490780 18340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "b5dc66531b4746aeb102e6dd5aca0153"
format_stamp: "Formatted at 2026-09-29 16:00:56 on dist-test-slave-b69v"
server_key: "5b438804cacb3dbf464c87df18f5f511"
server_key_iv: "94ffe4e7c5d70e92d3cbb28e13d82b32"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:56.497918 18340 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 16:00:56.517230 18355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:56.518826 18340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:00:56.519004 18340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "b5dc66531b4746aeb102e6dd5aca0153"
format_stamp: "Formatted at 2026-09-29 16:00:56 on dist-test-slave-b69v"
server_key: "5b438804cacb3dbf464c87df18f5f511"
server_key_iv: "94ffe4e7c5d70e92d3cbb28e13d82b32"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:56.519222 18340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:56.547410 18340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:56.551604 18340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:56.551920 18340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:56.552985 18340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:56.554478 18340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:56.554592 18340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:56.554740 18340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:56.554814 18340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:56.589226 18340 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:44039
I20260929 16:00:56.589260 18467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:44039 every 8 connection(s)
I20260929 16:00:56.590569 18340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:56.591986 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18340
I20260929 16:00:56.592191 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:00:56.592844 5588 external_mini_cluster.cc:1428] Setting key 7169a22ee0e117956c66adf532dfdf3b
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697656, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:56.613497 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.593399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:40937 (local address 127.5.117.62:46145)
0929 16:00:56.600414 (+ 7015us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.600419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.600433 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:00:56.600478 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.600481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.600560 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.600652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.602108 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.602893 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.604397 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.604841 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.604937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.608127 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.608132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.608136 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.608169 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.610544 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.611181 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.611187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.611190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.611313 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.611729 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.611735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.611738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.612001 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.612205 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.612882 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.613125 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6920,"thread_start_us":6779,"threads_started":1}
I20260929 16:00:56.614472 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.594114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.3:40937)
0929 16:00:56.595347 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.595411 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:56.598468 (+ 3057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:56.600893 (+ 2425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:56.600901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:56.601368 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:56.601974 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.601990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.603013 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.603016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.604276 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.604296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.605016 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.605021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.605230 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.605852 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:56.605872 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:56.607881 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:56.610676 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.610683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.610696 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.611067 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.611422 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.611427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.611430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.611623 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.612183 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:56.612190 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:56.612517 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 16:00:56.612799 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 16:00:56.613270 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":547,"threads_started":1}
I20260929 16:00:56.615808 18468 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:00:56.616173 18468 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:56.616879 18468 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:00:56.618330 17978 ts_manager.cc:194] Registered new tserver with Master: b5dc66531b4746aeb102e6dd5aca0153 (127.5.117.3:44039)
I20260929 16:00:56.619472 17978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:40937
I20260929 16:00:56.620298 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:56.622146 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18058
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:00:56.642308 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.623222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.5.117.1:33961)
0929 16:00:56.624086 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.624096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.624130 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:00:56.624204 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.624211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.624453 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.624623 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.626163 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.627261 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.629118 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.629963 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.630100 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.635373 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.635390 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.635410 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.635456 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.638843 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.639959 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.639966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.639968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.640202 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.640620 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.640629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.640631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.641003 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.641284 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.641574 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.641897 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":445,"threads_started":1}
I20260929 16:00:56.643834 18186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:56.735014 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18058
I20260929 16:00:56.753728 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18197
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:00:56.770974 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.754742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.5.117.2:44899)
0929 16:00:56.755356 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.755365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.755395 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:00:56.755467 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.755473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.755704 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.755850 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.757275 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.758487 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.759968 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.760680 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.760783 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.765555 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.765565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.765578 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.765623 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.768398 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.769049 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.769056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.769058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.769190 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.769512 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.769515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.769517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.769872 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.770133 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.770419 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.770651 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":316,"threads_started":1}
I20260929 16:00:56.772496 18329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:56.865973 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18197
I20260929 16:00:56.887225 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18340
Sep 29 16:00:56 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:00:56.903786 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.888363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.5.117.3:44039)
0929 16:00:56.889021 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.889029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.889061 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:00:56.889140 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.889147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.889394 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.889577 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.890913 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.891878 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.893421 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.893997 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.894151 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.898408 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.898419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.898434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.898472 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.901207 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.901795 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.901801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.901803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.901931 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.902212 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.902215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.902217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.902530 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.902804 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.903151 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.903434 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":337,"threads_started":1}
I20260929 16:00:56.905217 18463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:56.992552 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18340
I20260929 16:00:57.010092 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 17946
I20260929 16:00:57.011480 18008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:57.128706 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 17946
I20260929 16:00:57.156756 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46145
--webserver_interface=127.5.117.62
--webserver_port=40221
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46145
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:57.512382 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:57.512761 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:57.512921 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:57.520727 18480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:57.520850 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:57.520924 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:57.520979 18480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:57.521023 18480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:57.531443 18480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46145
--ranger_config_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=40221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:57.533193 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:57.534614 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:57.548069 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:57.548125 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:57.548126 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:57.548887 18480 server_base.cc:1034] running on GCE node
I20260929 16:00:57.549678 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:57.551194 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:57.552419 18480 hybrid_clock.cc:648] HybridClock initialized: now 1790697657552357 us; error 35 us; skew 500 ppm
I20260929 16:00:57.552788 18480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:57 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:57.562587 18480 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:00:57.564285 18480 webserver.cc:460] Webserver started at http://127.5.117.62:40221/ using document root <none> and password file <none>
I20260929 16:00:57.565012 18480 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:57.565141 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:57 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697657, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:57.635293 18480 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:00:57.639575 18496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:57.641403 18480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:00:57.641633 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "9043f2d24f924301bb5d2a1f5f0d41cb"
format_stamp: "Formatted at 2026-09-29 16:00:53 on dist-test-slave-b69v"
server_key: "b3d0d3cf6a21f89b81de2d58be7b3d9c"
server_key_iv: "b8eb4466e13c296e08bb5efac7f03959"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:57.643049 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:57.675817 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:57.681435 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:57.681878 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:57.703840 18480 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46145
I20260929 16:00:57.703855 18547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46145 every 8 connection(s)
I20260929 16:00:57.705550 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:57.713227 18548 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:00:57.714067 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18480
I20260929 16:00:57.715034 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:33961
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:46145
--webserver_port=41279
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:57.717749 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Bootstrap starting.
I20260929 16:00:57.743909 18548 log.cc:824] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:57.754339 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:00:57.754916 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Bootstrap complete.
I20260929 16:00:57.764621 18548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:57.765579 18548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9043f2d24f924301bb5d2a1f5f0d41cb, State: Initialized, Role: FOLLOWER
I20260929 16:00:57.766319 18548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:57.766587 18548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:57.766721 18548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:57.766904 18548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:00:57.771009 18548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:57.771531 18548 leader_election.cc:302] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9043f2d24f924301bb5d2a1f5f0d41cb; no voters:
I20260929 16:00:57.772224 18548 leader_election.cc:288] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:00:57.772377 18552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:00:57.773717 18552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [term 2 LEADER]: Becoming Leader. State: Replica: 9043f2d24f924301bb5d2a1f5f0d41cb, State: Running, Role: LEADER
I20260929 16:00:57.774364 18552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } }
I20260929 16:00:57.774730 18548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:57.777329 18554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9043f2d24f924301bb5d2a1f5f0d41cb. Latest consensus state: current_term: 2 leader_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } } }
I20260929 16:00:57.777367 18553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9043f2d24f924301bb5d2a1f5f0d41cb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46145 } } }
I20260929 16:00:57.777956 18554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:57.777961 18553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:57.783164 18558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:57.800882 18558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:57.802804 18558 catalog_manager.cc:1259] Loaded cluster ID: f8e2c406ec6544498e4cb465f5c0b39e
I20260929 16:00:57.802971 18558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:57.807497 18558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:57.809077 18558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Loaded TSK: 0
I20260929 16:00:57.810724 18558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:00:57.818536 18548 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 16:00:58.230993 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:58.231549 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:58.231768 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:58.244263 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:58.244481 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:58.244735 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:00:58.261327 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:33961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=41279
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:58.263563 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:58.265764 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:58.291953 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:58.300495 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:58.305441 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:58.305801 18550 server_base.cc:1034] running on GCE node
I20260929 16:00:58.306821 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:58.309859 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:58.311280 18550 hybrid_clock.cc:648] HybridClock initialized: now 1790697658311194 us; error 78 us; skew 500 ppm
I20260929 16:00:58.311718 18550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:58 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:58.322692 18550 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:00:58.337249 18550 webserver.cc:460] Webserver started at http://127.5.117.1:41279/ using document root <none> and password file <none>
I20260929 16:00:58.338441 18550 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:58.338616 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:58 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:58.383821 18550 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.008s
I20260929 16:00:58.389734 18596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:58.391803 18550 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:00:58.392017 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9"
format_stamp: "Formatted at 2026-09-29 16:00:54 on dist-test-slave-b69v"
server_key: "98a920ed89a6d6697f31b97f1f2cde86"
server_key_iv: "e8848e0d6853313d66d00467267101f9"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:58.393201 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:58.410745 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:58.414939 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:58.415236 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:58.416378 18550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:58.417825 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:58.417927 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:58.418056 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:58.418147 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:58.479956 18550 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:33961
I20260929 16:00:58.480008 18709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:33961 every 8 connection(s)
I20260929 16:00:58.481716 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:58.491801 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18550
I20260929 16:00:58.492815 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:44899
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:46145
--webserver_port=45063
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:58 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697658, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:58.519362 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:58.485429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.1:50969)
0929 16:00:58.490288 (+ 4859us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:58.490350 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:58.493265 (+ 2915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:58.496019 (+ 2754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:58.496032 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:58.496837 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:58.497903 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:58.497935 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:58.500668 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:58.500678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:58.502999 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:58.503026 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:58.504609 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:58.504619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:58.504929 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:58.505873 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:58.505924 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:58.509138 (+ 3214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:58.514367 (+ 5229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:58.514377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:58.514391 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:58.515034 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:58.515631 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:58.515636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:58.515640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:58.515852 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:58.516702 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:58.516711 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:58.517029 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 16:00:58.517297 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 16:00:58.517882 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4553,"thread_start_us":4335,"threads_started":1}
I20260929 16:00:58.519364 18713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:58.485435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:50969 (local address 127.5.117.62:46145)
0929 16:00:58.489602 (+ 4167us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:58.489632 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:58.490870 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 16:00:58.494205 (+ 3335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:58.494229 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:58.494796 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:58.495229 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:58.498176 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:58.500402 (+ 2226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:58.503183 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:58.504327 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:58.504537 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:58.509408 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:58.509416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:58.509431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:58.509467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:58.514152 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:58.515265 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:58.515273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:58.515280 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:58.515500 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:58.515976 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:58.515981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:58.515984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:58.516553 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:58.516762 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:58.517549 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 16:00:58.517918 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3773,"thread_start_us":3487,"threads_started":1}
I20260929 16:00:58.523416 18710 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:00:58.523769 18710 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:58.524742 18710 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
W20260929 16:00:58.528084 18710 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:46145 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:00:58.528329 18710 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:58.528599 18710 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:00:58.529892 18710 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:58.530236 18710 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
W20260929 16:00:58.531368 18710 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:00:58.970227 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:58.970738 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:58.970976 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:58.981307 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:58.981511 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:58.981796 18714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:00:58.997217 18714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:44899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=45063
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:58.999428 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:59.001495 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:59.020254 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:59.020251 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:59.021447 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:59.021697 18714 server_base.cc:1034] running on GCE node
I20260929 16:00:59.022746 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:59.023957 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:59.027328 18714 hybrid_clock.cc:648] HybridClock initialized: now 1790697659026875 us; error 430 us; skew 500 ppm
I20260929 16:00:59.027688 18714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:59.031209 18714 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:00:59.062405 18714 webserver.cc:460] Webserver started at http://127.5.117.2:45063/ using document root <none> and password file <none>
I20260929 16:00:59.063841 18714 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:59.064069 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:59.117756 18714 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:00:59.121744 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:59.123623 18714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:00:59.123816 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "4112eb4246564248b819594c300d29ca"
format_stamp: "Formatted at 2026-09-29 16:00:55 on dist-test-slave-b69v"
server_key: "58a744ad5e86b6adcf37451764d78737"
server_key_iv: "a0395ea7db577caad0af09364477fd29"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:59.125366 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:59.143676 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:59.149214 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:59.149551 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:59.150841 18714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:59.152859 18714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:59.152992 18714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:59.153110 18714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:59.153214 18714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:59.232656 18714 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44899
I20260929 16:00:59.232663 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44899 every 8 connection(s)
I20260929 16:00:59.235078 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:59.237097 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18714
I20260929 16:00:59.237951 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.117.20:46069/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:44039
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:46145
--webserver_port=43713
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:00:59.277144 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:59.239530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:49179 (local address 127.5.117.62:46145)
0929 16:00:59.256467 (+ 16937us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:59.256474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:59.256501 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:00:59.256590 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:59.256596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:59.256694 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:59.256826 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:59.259285 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:59.260503 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:59.262992 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:59.263682 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:59.263808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:59.268726 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:59.268733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:59.268738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:59.268782 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:59.272736 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:59.273679 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:59.273685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:59.273689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:59.273854 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:59.274429 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:59.274434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:59.274437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:59.274848 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:59.275004 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:59.276499 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0929 16:00:59.276735 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16773,"thread_start_us":16659,"threads_started":1}
I20260929 16:00:59.279525 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:59.240165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.2:49179)
0929 16:00:59.250756 (+ 10591us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:59.250826 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:59.254524 (+ 3698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:59.257344 (+ 2820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:59.257359 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:59.258081 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:59.259014 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:59.259043 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:59.260711 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:59.260718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:59.262805 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:59.262830 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:59.264192 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:59.264201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:59.264492 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:59.265404 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:59.265446 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:59.268461 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:59.272893 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:59.272902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:59.272919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:59.273520 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:59.273997 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:59.274003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:59.274007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:59.274283 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:59.275108 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:59.275116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:59.275574 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 16:00:59.275955 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 16:00:59.278039 (+ 2084us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10144,"thread_start_us":9879,"threads_started":1}
I20260929 16:00:59.281879 18854 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:00:59.282258 18854 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:59.283245 18854 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:00:59.285869 18513 ts_manager.cc:194] Registered new tserver with Master: 4112eb4246564248b819594c300d29ca (127.5.117.2:44899)
I20260929 16:00:59.288465 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:49179
I20260929 16:00:59.531631 18710 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:59.532300 18710 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:00:59.534307 18513 ts_manager.cc:194] Registered new tserver with Master: 3cf9a4cb02a24b0ba6c5303ccd57eac9 (127.5.117.1:33961)
I20260929 16:00:59.535709 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:50969
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
W20260929 16:00:59.768541 18858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:59.769106 18858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:59.769345 18858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:59.782344 18858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:59.782549 18858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:59.782812 18858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:00:59.799840 18858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.117.20:46069/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:44039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=43713
--tserver_master_addrs=127.5.117.62:46145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:00:59.802275 18858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:59.804298 18858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:59.822924 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:59.831482 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:59.833196 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:59.833390 18858 server_base.cc:1034] running on GCE node
I20260929 16:00:59.834400 18858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:59.835942 18858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:59.837280 18858 hybrid_clock.cc:648] HybridClock initialized: now 1790697659837210 us; error 56 us; skew 500 ppm
I20260929 16:00:59.837725 18858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:59.842693 18858 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:00:59.844924 18858 webserver.cc:460] Webserver started at http://127.5.117.3:43713/ using document root <none> and password file <none>
I20260929 16:00:59.846026 18858 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:59.846192 18858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:59.890125 18858 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.012s sys 0.000s
I20260929 16:00:59.894375 18875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:59.897680 18858 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 16:00:59.898169 18858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "b5dc66531b4746aeb102e6dd5aca0153"
format_stamp: "Formatted at 2026-09-29 16:00:56 on dist-test-slave-b69v"
server_key: "5b438804cacb3dbf464c87df18f5f511"
server_key_iv: "94ffe4e7c5d70e92d3cbb28e13d82b32"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:59.899770 18858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 16:00:59 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for HTTP/127.5.117.20@KRBTEST.COM
I20260929 16:00:59.948864 18858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:59.957417 18858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:59.957880 18858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:59.967862 18858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:59.985503 18858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:59.985800 18858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:59.986017 18858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:59.986157 18858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:00.204495 18858 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:44039
I20260929 16:01:00.204514 18988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:44039 every 8 connection(s)
I20260929 16:01:00.206519 18858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:00.213575 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 18858
Sep 29 16:01:00 dist-test-slave-b69v krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697659, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:00.257445 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.212651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:46607 (local address 127.5.117.62:46145)
0929 16:01:00.222312 (+ 9661us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.222319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.229188 (+ 6869us) server_negotiation.cc:407] Connection header received
0929 16:01:00.229302 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.229308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.229442 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.229575 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:00.234724 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.236342 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.241298 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.241973 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.242093 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.248236 (+ 6143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:00.248243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:00.248248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:00.248455 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:00.252638 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.253800 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.253807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.253812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.254125 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.254762 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.254772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.254777 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.255227 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:00.255434 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.256915 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.257129 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9494,"thread_start_us":18234,"threads_started":1}
I20260929 16:01:00.259115 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.211193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46145 (local address 127.5.117.3:46607)
0929 16:01:00.225421 (+ 14228us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.225480 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.228462 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.230129 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.230143 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.231078 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:00.232180 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.232216 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.238855 (+ 6639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.238861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.241099 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.241130 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.242693 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.242704 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.243410 (+ 706us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.244782 (+ 1372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:00.244816 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:00.247884 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:00.252860 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:00.252871 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:00.252894 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:00.253620 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:00.254303 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:00.254308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:00.254311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:00.254624 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:00.255502 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:00.255516 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:00.256354 (+ 838us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.256778 (+ 424us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.257442 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12727,"thread_start_us":12716,"threads_started":1}
I20260929 16:01:00.261039 18989 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46145
I20260929 16:01:00.261487 18989 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:00.262399 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.231135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.5.117.62:46145)
0929 16:01:00.231907 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.231915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.232018 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:01:00.232118 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.232124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.232242 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.232380 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:00.234715 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.235978 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.239321 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.239774 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.239930 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.243719 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:00.243724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:00.243727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:00.243765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:00.247489 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.259880 (+ 12391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.259887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.259890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.260086 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.260624 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.260630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.260634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.260992 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:00.261287 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.261728 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.261984 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":361,"threads_started":1}
I20260929 16:01:00.262605 18989 heartbeater.cc:511] Master 127.5.117.62:46145 requested a full tablet report, sending...
I20260929 16:01:00.264642 18513 ts_manager.cc:194] Registered new tserver with Master: b5dc66531b4746aeb102e6dd5aca0153 (127.5.117.3:44039)
I20260929 16:01:00.265877 18513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:46607
I20260929 16:01:00.265901 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:00.289089 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.274936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.5.117.62:46145)
0929 16:01:00.275228 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.275233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.275556 (+ 323us) server_negotiation.cc:407] Connection header received
0929 16:01:00.275727 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.275731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.275816 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.275964 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:00.277879 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.278708 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.280208 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.280644 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.280737 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.283408 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:00.283413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:00.283416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:00.283456 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:00.286210 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.286989 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.286996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.286999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.287256 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.287757 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.287762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.287764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.288149 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:00.288345 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.288630 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.288799 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 16:01:00.294725 18854 heartbeater.cc:503] Master 127.5.117.62:46145 was elected leader, sending a full tablet report...
I20260929 16:01:00.294679 18513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:00.304088 18513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:00.339318 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.319409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:44899 (local address 127.0.0.1:54118)
0929 16:01:00.329212 (+ 9803us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.329272 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.329535 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.331178 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.331186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.331218 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:00.331763 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.331782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.333867 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.333875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.336576 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.336596 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.338124 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.338130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.338337 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.338400 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.338547 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.338758 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9396,"mutex_wait_us":2,"thread_start_us":11024,"threads_started":1}
I20260929 16:01:00.340196 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.319789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:33961 (local address 127.0.0.1:49942)
0929 16:01:00.330742 (+ 10953us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.330761 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.330880 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.332569 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.332574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.332599 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:00.332994 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.333003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.334846 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.334851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.337034 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.337048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.339298 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.339310 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.339493 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.339587 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.339667 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.339790 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10842,"thread_start_us":12329,"threads_started":1}
I20260929 16:01:00.340648 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.319755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.5.117.2:44899)
0929 16:01:00.330416 (+ 10661us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.330429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.330470 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:01:00.330599 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.330611 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.330934 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.331177 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:00.331979 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.333730 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.336753 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.337898 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.338063 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.339965 (+ 1902us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.340075 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.340283 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10455,"thread_start_us":10264,"threads_started":1}
I20260929 16:01:00.340855 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.320764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.5.117.1:33961)
0929 16:01:00.331824 (+ 11060us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.331836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.331877 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:01:00.332008 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.332020 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.332326 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.333157 (+ 831us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:00.333453 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.334680 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.337205 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.338724 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.338857 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.339096 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.340273 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.340458 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10845,"thread_start_us":10647,"threads_started":1}
I20260929 16:01:00.340902 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.320231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:44039 (local address 127.0.0.1:33672)
0929 16:01:00.332525 (+ 12294us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.332544 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.332678 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.333338 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.333343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.333364 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:00.333788 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.333800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.335769 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.335780 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.337801 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.337815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.340078 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.340084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.340304 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.340349 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.340430 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.340553 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12157,"thread_start_us":8504,"threads_started":1}
I20260929 16:01:00.342232 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.321294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.5.117.3:44039)
0929 16:01:00.331236 (+ 9942us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.331248 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.332805 (+ 1557us) server_negotiation.cc:407] Connection header received
0929 16:01:00.332909 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.332916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.333124 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.333326 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:00.334011 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.335604 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.338003 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.339505 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.339638 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.339873 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.341826 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.341955 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9748,"thread_start_us":9579,"threads_started":1}
I20260929 16:01:00.345326 18789 tablet_service.cc:1467] Processing CreateTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 (DEFAULT_TABLE table=test-table [id=8986c909979c4200b526278081a84728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:00.345329 18645 tablet_service.cc:1467] Processing CreateTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 (DEFAULT_TABLE table=test-table [id=8986c909979c4200b526278081a84728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:00.345683 18924 tablet_service.cc:1467] Processing CreateTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 (DEFAULT_TABLE table=test-table [id=8986c909979c4200b526278081a84728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:00.346973 18924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9479dc7d5fc4c2f8df0deafee207cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:00.347023 18789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9479dc7d5fc4c2f8df0deafee207cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:00.347016 18645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9479dc7d5fc4c2f8df0deafee207cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:00.360149 19017 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Bootstrap starting.
I20260929 16:01:00.362941 19018 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Bootstrap starting.
I20260929 16:01:00.363693 19019 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Bootstrap starting.
I20260929 16:01:00.365183 19017 tablet_bootstrap.cc:654] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:00.366585 19017 log.cc:824] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:00.366756 19018 tablet_bootstrap.cc:654] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:00.367388 19019 tablet_bootstrap.cc:654] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:00.368053 19018 log.cc:824] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:00.368610 19019 log.cc:824] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:00.370352 19017 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: No bootstrap required, opened a new log
I20260929 16:01:00.370628 19017 ts_tablet_manager.cc:1397] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 16:01:00.371567 19018 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: No bootstrap required, opened a new log
I20260929 16:01:00.371775 19019 tablet_bootstrap.cc:492] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: No bootstrap required, opened a new log
I20260929 16:01:00.371838 19018 ts_tablet_manager.cc:1397] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 16:01:00.372027 19019 ts_tablet_manager.cc:1397] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 16:01:00.377959 19017 raft_consensus.cc:348] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.378304 19017 raft_consensus.cc:374] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:00.378427 19017 raft_consensus.cc:729] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dc66531b4746aeb102e6dd5aca0153, State: Initialized, Role: FOLLOWER
I20260929 16:01:00.379109 19017 consensus_queue.cc:260] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.381541 18989 heartbeater.cc:503] Master 127.5.117.62:46145 was elected leader, sending a full tablet report...
I20260929 16:01:00.382189 19017 ts_tablet_manager.cc:1428] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260929 16:01:00.382791 19018 raft_consensus.cc:348] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.382791 19019 raft_consensus.cc:348] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.383201 19019 raft_consensus.cc:374] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:00.383199 19018 raft_consensus.cc:374] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:00.383306 19019 raft_consensus.cc:729] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf9a4cb02a24b0ba6c5303ccd57eac9, State: Initialized, Role: FOLLOWER
I20260929 16:01:00.383311 19018 raft_consensus.cc:729] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4112eb4246564248b819594c300d29ca, State: Initialized, Role: FOLLOWER
I20260929 16:01:00.383900 19018 consensus_queue.cc:260] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.383903 19019 consensus_queue.cc:260] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.385614 18710 heartbeater.cc:503] Master 127.5.117.62:46145 was elected leader, sending a full tablet report...
I20260929 16:01:00.385900 19018 ts_tablet_manager.cc:1428] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:01:00.386191 19019 ts_tablet_manager.cc:1428] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
W20260929 16:01:00.464401 18990 tablet.cc:2369] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:00.486953 18711 tablet.cc:2369] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:00.487834 18855 tablet.cc:2369] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:00.612005 19025 raft_consensus.cc:482] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:00.612416 19025 raft_consensus.cc:504] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.614359 19025 leader_election.cc:288] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cf9a4cb02a24b0ba6c5303ccd57eac9 (127.5.117.1:33961), b5dc66531b4746aeb102e6dd5aca0153 (127.5.117.3:44039)
I20260929 16:01:00.624112 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.614907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:56687 (local address 127.5.117.1:33961)
0929 16:01:00.615204 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.615212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.616100 (+ 888us) server_negotiation.cc:407] Connection header received
0929 16:01:00.616418 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.616427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.616572 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.616671 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:00.617571 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.618382 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.621633 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.622910 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.623046 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.623176 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.623639 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.623787 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:01:00.624228 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.614884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:33961 (local address 127.5.117.2:56687)
0929 16:01:00.615930 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.615956 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.616207 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.616881 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.616886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.616988 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:00.617427 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.617439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.618537 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.618545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.621385 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.621405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.623322 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.623327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.623448 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.623475 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.623597 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.623736 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":205,"spinlock_wait_cycles":47872,"thread_start_us":469,"threads_started":1}
I20260929 16:01:00.624473 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.615080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:58075 (local address 127.5.117.3:44039)
0929 16:01:00.615314 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.615339 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.616207 (+ 868us) server_negotiation.cc:407] Connection header received
0929 16:01:00.616458 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.616469 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.616539 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.616647 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:00.617531 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.618509 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.621606 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.622901 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.623046 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.623167 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.623983 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.624162 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:01:00.625094 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.614994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:44039 (local address 127.5.117.2:58075)
0929 16:01:00.616023 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.616099 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:00.616297 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:00.616672 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:00.616676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:00.616703 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:00.617261 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.617277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.618622 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.618627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.621385 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:00.621405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.623144 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.623153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:00.623785 (+ 632us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.623819 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:01:00.624656 (+ 837us) client_negotiation.cc:240] Negotiation successful
0929 16:01:00.624787 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":149,"thread_start_us":680,"threads_started":1}
I20260929 16:01:00.625727 18665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9479dc7d5fc4c2f8df0deafee207cc0" candidate_uuid: "4112eb4246564248b819594c300d29ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" is_pre_election: true
I20260929 16:01:00.626207 18944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9479dc7d5fc4c2f8df0deafee207cc0" candidate_uuid: "4112eb4246564248b819594c300d29ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dc66531b4746aeb102e6dd5aca0153" is_pre_election: true
I20260929 16:01:00.626528 18665 raft_consensus.cc:2454] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4112eb4246564248b819594c300d29ca in term 0.
I20260929 16:01:00.626804 18944 raft_consensus.cc:2454] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4112eb4246564248b819594c300d29ca in term 0.
I20260929 16:01:00.627537 18742 leader_election.cc:302] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3cf9a4cb02a24b0ba6c5303ccd57eac9, 4112eb4246564248b819594c300d29ca, b5dc66531b4746aeb102e6dd5aca0153; no voters:
I20260929 16:01:00.628229 19025 raft_consensus.cc:2781] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:00.628485 19025 raft_consensus.cc:482] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:00.628626 19025 raft_consensus.cc:3037] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:00.631529 19025 raft_consensus.cc:504] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.632295 19025 leader_election.cc:288] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [CANDIDATE]: Term 1 election: Requested vote from peers 3cf9a4cb02a24b0ba6c5303ccd57eac9 (127.5.117.1:33961), b5dc66531b4746aeb102e6dd5aca0153 (127.5.117.3:44039)
I20260929 16:01:00.632743 18665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9479dc7d5fc4c2f8df0deafee207cc0" candidate_uuid: "4112eb4246564248b819594c300d29ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9"
I20260929 16:01:00.632750 18944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9479dc7d5fc4c2f8df0deafee207cc0" candidate_uuid: "4112eb4246564248b819594c300d29ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dc66531b4746aeb102e6dd5aca0153"
I20260929 16:01:00.632953 18944 raft_consensus.cc:3037] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:00.632963 18665 raft_consensus.cc:3037] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:00.635269 18944 raft_consensus.cc:2454] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4112eb4246564248b819594c300d29ca in term 1.
I20260929 16:01:00.635311 18665 raft_consensus.cc:2454] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4112eb4246564248b819594c300d29ca in term 1.
I20260929 16:01:00.635679 18742 leader_election.cc:302] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4112eb4246564248b819594c300d29ca, b5dc66531b4746aeb102e6dd5aca0153; no voters:
I20260929 16:01:00.635963 19025 raft_consensus.cc:2781] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:00.636521 19025 raft_consensus.cc:686] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 1 LEADER]: Becoming Leader. State: Replica: 4112eb4246564248b819594c300d29ca, State: Running, Role: LEADER
I20260929 16:01:00.637033 19025 consensus_queue.cc:237] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } }
I20260929 16:01:00.642254 18513 catalog_manager.cc:5495] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4112eb4246564248b819594c300d29ca (127.5.117.2). New cstate: current_term: 1 leader_uuid: "4112eb4246564248b819594c300d29ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4112eb4246564248b819594c300d29ca" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 } health_report { overall_health: UNKNOWN } } }
I20260929 16:01:00.710460 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.703059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.5.117.2:44899)
0929 16:01:00.703295 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.703301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.703324 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:01:00.703629 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.703635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.703765 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.703960 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:00.704382 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.705559 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.707314 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.707980 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.708134 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.708719 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:00.709526 (+ 807us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:00.709703 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.710056 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.710152 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:01:00.720739 18944 raft_consensus.cc:1261] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 1 FOLLOWER]: Refusing update from remote peer 4112eb4246564248b819594c300d29ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:00.720739 18665 raft_consensus.cc:1261] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 1 FOLLOWER]: Refusing update from remote peer 4112eb4246564248b819594c300d29ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:00.722069 19025 consensus_queue.cc:1035] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cf9a4cb02a24b0ba6c5303ccd57eac9" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:00.722434 19028 consensus_queue.cc:1035] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5dc66531b4746aeb102e6dd5aca0153" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 44039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:00.736753 19031 mvcc.cc:205] Tried to move back new op lower bound from 7334697618298150912 to 7334697617973460992. Current Snapshot: MvccSnapshot[applied={T|T < 7334697618298150912}]
I20260929 16:01:00.737703 19032 mvcc.cc:205] Tried to move back new op lower bound from 7334697618298150912 to 7334697617973460992. Current Snapshot: MvccSnapshot[applied={T|T < 7334697618298150912}]
I20260929 16:01:00.752199 18513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:00.752542 18513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:00.756771 18513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9043f2d24f924301bb5d2a1f5f0d41cb: Sending DeleteTablet for 3 replicas of tablet e9479dc7d5fc4c2f8df0deafee207cc0
I20260929 16:01:00.758028 18645 tablet_service.cc:1514] Processing DeleteTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:00 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:49942
I20260929 16:01:00.758019 18924 tablet_service.cc:1514] Processing DeleteTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:00 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:33672
I20260929 16:01:00.758060 18789 tablet_service.cc:1514] Processing DeleteTablet for tablet e9479dc7d5fc4c2f8df0deafee207cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:00 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:54118
I20260929 16:01:00.765061 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18550
I20260929 16:01:00.774590 19041 tablet_replica.cc:330] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: stopping tablet replica
I20260929 16:01:00.775211 19041 raft_consensus.cc:2229] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:00.776134 19041 raft_consensus.cc:2258] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:00.780922 19042 tablet_replica.cc:330] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: stopping tablet replica
I20260929 16:01:00.781451 19042 raft_consensus.cc:2229] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:00.781955 19041 ts_tablet_manager.cc:1905] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:00.782050 19042 raft_consensus.cc:2258] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:00.784195 19042 ts_tablet_manager.cc:1905] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:00.792239 19042 ts_tablet_manager.cc:1918] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:00.792950 19041 ts_tablet_manager.cc:1918] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:00.793134 19041 log.cc:1196] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/e9479dc7d5fc4c2f8df0deafee207cc0
I20260929 16:01:00.793059 19043 tablet_replica.cc:330] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: stopping tablet replica
I20260929 16:01:00.793634 19041 ts_tablet_manager.cc:1939] T e9479dc7d5fc4c2f8df0deafee207cc0 P 4112eb4246564248b819594c300d29ca: Deleting consensus metadata
I20260929 16:01:00.793807 19043 raft_consensus.cc:2229] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:00.794549 19043 raft_consensus.cc:2258] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:00.795423 18499 catalog_manager.cc:4841] TS 4112eb4246564248b819594c300d29ca (127.5.117.2:44899): tablet e9479dc7d5fc4c2f8df0deafee207cc0 (table test-table [id=8986c909979c4200b526278081a84728]) successfully deleted
I20260929 16:01:00.795962 19042 log.cc:1196] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/e9479dc7d5fc4c2f8df0deafee207cc0
I20260929 16:01:00.796504 19042 ts_tablet_manager.cc:1939] T e9479dc7d5fc4c2f8df0deafee207cc0 P b5dc66531b4746aeb102e6dd5aca0153: Deleting consensus metadata
I20260929 16:01:00.797547 19043 ts_tablet_manager.cc:1905] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:00.798038 18497 catalog_manager.cc:4841] TS b5dc66531b4746aeb102e6dd5aca0153 (127.5.117.3:44039): tablet e9479dc7d5fc4c2f8df0deafee207cc0 (table test-table [id=8986c909979c4200b526278081a84728]) successfully deleted
I20260929 16:01:00.804908 19043 ts_tablet_manager.cc:1918] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:00.805138 19043 log.cc:1196] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/e9479dc7d5fc4c2f8df0deafee207cc0
I20260929 16:01:00.805781 19043 ts_tablet_manager.cc:1939] T e9479dc7d5fc4c2f8df0deafee207cc0 P 3cf9a4cb02a24b0ba6c5303ccd57eac9: Deleting consensus metadata
I20260929 16:01:00.807423 18500 catalog_manager.cc:4841] TS 3cf9a4cb02a24b0ba6c5303ccd57eac9 (127.5.117.1:33961): tablet e9479dc7d5fc4c2f8df0deafee207cc0 (table test-table [id=8986c909979c4200b526278081a84728]) successfully deleted
I20260929 16:01:00.811251 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.767288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.5.117.1:33961)
0929 16:01:00.767543 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.767552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.780650 (+ 13098us) server_negotiation.cc:407] Connection header received
0929 16:01:00.780877 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.780883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.781009 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.781149 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:00.794695 (+ 13546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.796011 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.798009 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.798715 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.798819 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.802828 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:00.802847 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:00.802868 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:00.802940 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:00.808125 (+ 5185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.808838 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.808849 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.808853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.809041 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.809430 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.809435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.809438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.809954 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:00.810288 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.810568 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.810814 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:01:00.812000 18705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:00.913122 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18550
I20260929 16:01:00.934094 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18714
I20260929 16:01:00.951957 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.934973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.5.117.2:44899)
0929 16:01:00.935178 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:00.935183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:00.935446 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:01:00.935628 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:00.935634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:00.935719 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:00.935820 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:00.937768 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.938664 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.940609 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:00.941222 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:00.941324 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:00.945100 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:00.945120 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:00.945139 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:00.945200 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:00.948957 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.949678 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.949687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.949689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.949846 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:00.950327 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:00.950330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:00.950332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:00.950709 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:00.951050 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:00.951316 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:01:00.951568 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:01:00.952832 18849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:01.078063 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18714
I20260929 16:01:01.100445 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18858
I20260929 16:01:01.115978 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:01.101341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.5.117.3:44039)
0929 16:01:01.101550 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:01.101556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:01.101579 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:01:01.101753 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:01.101758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:01.101839 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:01.101953 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:01.103452 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:01.104350 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:01.105846 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:01.106266 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:01.106415 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:01.109511 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:01.109524 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:01.109545 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:01.109617 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:01.112890 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:01.113672 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:01.113680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:01.113682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:01.113831 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:01.114216 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:01.114223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:01.114226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:01.114645 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:01.114976 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:01.115331 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:01:01.115574 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:01:01.116863 18984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:01.211191 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18858
I20260929 16:01:01.229499 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 18480
I20260929 16:01:01.230715 18543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:01.355878 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 18480
I20260929 16:01:01.383684 5588 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:01:01.731132 5588 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:01:01.731371 5588 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:01:02.088284 5588 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:01:02.088 UTC [17532] LOG: received smart shutdown request
2026-09-29 16:01:02.092 UTC [17532] LOG: background worker "logical replication launcher" (PID 17541) exited with exit code 1
2026-09-29 16:01:02.092 UTC [17536] LOG: shutting down
2026-09-29 16:01:02.296 UTC [17532] LOG: database system is shut down
2026-09-29T16:01:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58037 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:01:02 dist-test-slave-b69v krb5kdc[19057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:01:02 dist-test-slave-b69v krb5kdc[19057](info): set up 2 sockets
Sep 29 16:01:02 dist-test-slave-b69v krb5kdc[19057](info): commencing operation
krb5kdc: starting...
W20260929 16:01:03.560846 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:01:03 dist-test-slave-b69v krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:01:03.862123 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35943
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:04.264566 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:04.264932 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:04.265117 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:04.273007 19070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:04.273145 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:04.273231 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:04.273330 19070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:04.273409 19070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:04.284055 19070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:04.285768 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:04.287323 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:04.304833 19075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:04.304823 19078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:04.304811 19076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:04.306464 19070 server_base.cc:1034] running on GCE node
I20260929 16:01:04.307616 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:04.309599 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:04.310853 19070 hybrid_clock.cc:648] HybridClock initialized: now 1790697664310801 us; error 50 us; skew 500 ppm
I20260929 16:01:04.311285 19070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:04 dist-test-slave-b69v krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:04.319525 19070 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:04.321553 19070 webserver.cc:460] Webserver started at http://127.5.117.62:44201/ using document root <none> and password file <none>
I20260929 16:01:04.322389 19070 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:04.322504 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:04.322805 19070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:04.327016 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "19ea96b60c5840778174bd6f4154e4a5"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "347e5e908b3d9e2c0760a80a734ccb3f"
server_key_iv: "754286ba6a6ff2bb5d1a700cf294b1e9"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.327934 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "19ea96b60c5840778174bd6f4154e4a5"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "347e5e908b3d9e2c0760a80a734ccb3f"
server_key_iv: "754286ba6a6ff2bb5d1a700cf294b1e9"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.333981 19070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:01:04.338721 19085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:04.340812 19070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:01:04.340991 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "19ea96b60c5840778174bd6f4154e4a5"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "347e5e908b3d9e2c0760a80a734ccb3f"
server_key_iv: "754286ba6a6ff2bb5d1a700cf294b1e9"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.341243 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:04.357300 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:04.363333 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:04.363723 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:04.387336 19070 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:35943
I20260929 16:01:04.387377 19136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:35943 every 8 connection(s)
I20260929 16:01:04.389060 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:04.393584 19137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:04.397125 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19070
I20260929 16:01:04.397409 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:01:04.398164 5588 external_mini_cluster.cc:1428] Setting key 1e5474baa117b4062d4a82205966e115
I20260929 16:01:04.401137 19137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:01:04 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.117.61@KRBTEST.COM; defaulting to no policy
I20260929 16:01:04.426692 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.405036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35943 (local address 127.0.0.1:39002)
0929 16:01:04.406504 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.406542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:04.408563 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:04.409435 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:04.409443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:04.410251 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:04.410623 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:04.410641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:04.412441 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:04.412451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:04.413609 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:04.413623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:04.414626 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:04.414633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:04.414813 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:04.415600 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:04.415626 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:04.418518 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:04.421974 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:04.421986 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:04.422000 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:04.422400 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:04.422815 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:04.422821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:04.422827 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:04.422995 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:04.423803 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:04.423816 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:04.424089 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 16:01:04.424383 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 16:01:04.425095 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":206,"thread_start_us":1010,"threads_started":1}
I20260929 16:01:04.426714 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.405032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.5.117.62:35943)
0929 16:01:04.406145 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:04.406163 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:04.406705 (+ 542us) server_negotiation.cc:407] Connection header received
0929 16:01:04.408948 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:04.408955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:04.409193 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:04.409431 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:04.410836 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:04.412260 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:04.413769 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:04.414351 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:04.414572 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:04.418767 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:04.418776 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:04.418792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:04.418821 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:04.421825 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:04.422535 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:04.422545 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:04.422549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:04.422701 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:04.423101 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:04.423109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:04.423113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:04.423665 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:04.423896 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:04.424528 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 16:01:04.424977 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":730,"thread_start_us":469,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.61@KRBTEST.COM" created.
I20260929 16:01:04.430205 19137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:01:04.431586 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.431027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55280)
0929 16:01:04.431250 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.431413 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 16:01:04.431860 19086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
I20260929 16:01:04.433748 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.433253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55282)
0929 16:01:04.433446 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.433548 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:01:04.434015 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:32907: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
I20260929 16:01:04.463685 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:32907
--webserver_interface=127.5.117.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:04.483494 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } attempt: 1
I20260929 16:01:04.485163 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.484591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55290)
0929 16:01:04.484850 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.484974 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 16:01:04.486383 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.485974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55304)
0929 16:01:04.486171 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.486242 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 16:01:04.486680 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:32907: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
I20260929 16:01:04.555073 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } attempt: 2
I20260929 16:01:04.556783 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.556206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55318)
0929 16:01:04.556493 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.556608 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:01:04.557933 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.557589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55330)
0929 16:01:04.557725 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.557807 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:01:04.558208 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:32907: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
I20260929 16:01:04.649530 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } attempt: 3
I20260929 16:01:04.651170 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.650557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55338)
0929 16:01:04.650829 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.650960 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 16:01:04.652417 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.651936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55342)
0929 16:01:04.652150 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.652236 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:01:04.652635 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:32907: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
I20260929 16:01:04.795959 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } attempt: 4
I20260929 16:01:04.797492 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.796863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55348)
0929 16:01:04.797220 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.797318 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260929 16:01:04.798532 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.798149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55364)
0929 16:01:04.798296 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.798391 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:01:04.798698 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:32907: Network error: Client connection negotiation failed: client connection to 127.5.117.61:32907: connect: Connection refused (error 111)
W20260929 16:01:04.855701 19144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:04.856200 19144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:04.856374 19144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:04.864464 19144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:04.864640 19144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:04.864750 19144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:04.864856 19144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:04.864943 19144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:04.876256 19144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:32907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.117.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:04.878048 19144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:04.879640 19144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:04.894975 19150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:04.895308 19152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:04.894975 19149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:04.896663 19144 server_base.cc:1034] running on GCE node
I20260929 16:01:04.897639 19144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:04.899456 19144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:04.900704 19144 hybrid_clock.cc:648] HybridClock initialized: now 1790697664900636 us; error 60 us; skew 500 ppm
I20260929 16:01:04.901119 19144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:04 dist-test-slave-b69v krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:04.904990 19144 init.cc:377] Logged in from keytab as kudu/127.5.117.61@KRBTEST.COM (short username kudu)
I20260929 16:01:04.906776 19144 webserver.cc:460] Webserver started at http://127.5.117.61:36641/ using document root <none> and password file <none>
I20260929 16:01:04.907624 19144 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:04.907761 19144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:04.908301 19144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:04.911900 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data/instance:
uuid: "3e283348c28543cda9db79036c8a4fe1"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "8d1f3d93d59b4c21be999dc14cee5d3d"
server_key_iv: "d25f59d7b7cf6f8e4695529738ab4a63"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.912891 19144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal/instance:
uuid: "3e283348c28543cda9db79036c8a4fe1"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "8d1f3d93d59b4c21be999dc14cee5d3d"
server_key_iv: "d25f59d7b7cf6f8e4695529738ab4a63"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.919031 19144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:01:04.923292 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:04.924942 19144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:04.925148 19144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "3e283348c28543cda9db79036c8a4fe1"
format_stamp: "Formatted at 2026-09-29 16:01:04 on dist-test-slave-b69v"
server_key: "8d1f3d93d59b4c21be999dc14cee5d3d"
server_key_iv: "d25f59d7b7cf6f8e4695529738ab4a63"
server_key_version: "encryptionkey@0"
I20260929 16:01:04.925384 19144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:04.950104 19144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:04.955114 19144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:04.955446 19144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:04.977504 19144 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.61:32907
I20260929 16:01:04.977536 19210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.61:32907 every 8 connection(s)
I20260929 16:01:04.979226 19144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
I20260929 16:01:04.983644 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:04.985019 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19144
I20260929 16:01:04.985265 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-1/wal/instance
I20260929 16:01:04.985740 5588 external_mini_cluster.cc:1428] Setting key a73517b9ffb1660b94b3b7eb66c47717
I20260929 16:01:04.991284 19211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.117.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.60@KRBTEST.COM" created.
I20260929 16:01:05.016855 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.994292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.5.117.62:35943)
0929 16:01:04.994943 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:04.994953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:04.995944 (+ 991us) server_negotiation.cc:407] Connection header received
0929 16:01:04.998679 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:04.998689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:04.998809 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:04.998980 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.001087 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.002430 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.004349 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.005116 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.005270 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.009599 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.009608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.009615 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.009684 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.013207 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.014052 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.014057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.014059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.014190 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.014548 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.014551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.014553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.014901 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.015059 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.016076 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.016363 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":327,"threads_started":1}
I20260929 16:01:05.017988 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.994875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35943 (local address 127.0.0.1:39014)
0929 16:01:04.995750 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.995807 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:04.998302 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:04.999327 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:04.999339 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.000146 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.000850 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.000871 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.002646 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.002651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.004042 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.004135 (+ 93us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.005365 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.005371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.005634 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.006470 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.006513 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.009321 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.013356 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.013364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.013381 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.013929 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.014280 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.014283 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.014285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.014454 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.015035 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.015041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.015608 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.015899 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.016519 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":384,"threads_started":1}
I20260929 16:01:05.019927 19211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:01:05.035015 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.020635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55372)
0929 16:01:05.020837 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.020873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.020957 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.022101 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.022105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.022539 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.022670 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.022677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.023939 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.023946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.024751 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.024761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.025623 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.025628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.025891 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.026554 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.026570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.028919 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.032373 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.032379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.032385 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.032787 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.033181 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.033186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.033189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.033314 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.033930 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.033939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.034104 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.034242 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.034542 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:01:05.035054 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.021032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.5.117.61:32907)
0929 16:01:05.021705 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.021710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.021735 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:01:05.021772 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.021777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.021961 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.022112 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.022805 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.023806 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.024868 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.025457 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.025607 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.029126 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.029137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.029152 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.029182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.032242 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.032913 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.032922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.032930 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.033057 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.033409 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.033416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.033420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.033816 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.034041 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.034376 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.034633 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":533,"thread_start_us":387,"threads_started":1}
I20260929 16:01:05.038358 19211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:01:05.039623 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.039190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55680)
0929 16:01:05.039359 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.039468 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:01:05.039881 19162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.041503 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.041194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55686)
0929 16:01:05.041334 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.041402 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:01:05.041719 19211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
I20260929 16:01:05.049525 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:38239
--webserver_interface=127.5.117.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:05.091048 19211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 1
I20260929 16:01:05.092479 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.091919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55690)
0929 16:01:05.092170 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.092288 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:01:05.093555 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.093190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55692)
0929 16:01:05.093340 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.093434 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:01:05.093789 19211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.097934 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } attempt: 5
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:01:05.112636 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.098650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.5.117.61:32907)
0929 16:01:05.098831 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.098838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.098899 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:01:05.099148 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.099152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.099220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.099325 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.100313 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.101055 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.102298 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.102796 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.102944 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.107049 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.107054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.107057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.107090 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.109964 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.110735 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.110739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.110742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.110867 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.111281 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.111285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.111287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.111571 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.111691 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.112224 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.112411 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:01:05.112807 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.098595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55374)
0929 16:01:05.098796 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.098816 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.098981 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.099372 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.099379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.099896 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.100166 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.100179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.101195 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.101201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.102158 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.102170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.102949 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.102955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.103158 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.103839 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.103859 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.106851 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.110094 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.110099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.110103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.110626 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.110971 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.110976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.110979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.111171 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.111685 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.111692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.111890 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.112127 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.112444 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:01:05.113974 19137 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:01:05.115164 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.114657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55708)
0929 16:01:05.114850 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.114954 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:01:05.116222 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.115812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55710)
0929 16:01:05.115958 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.116077 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:01:05.116438 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.162073 19211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 2
I20260929 16:01:05.163431 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.162901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55724)
0929 16:01:05.163122 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.163229 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:01:05.164557 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.164217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55740)
0929 16:01:05.164363 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.164438 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:01:05.164834 19211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.167690 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 1
I20260929 16:01:05.169193 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.168586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55748)
0929 16:01:05.168827 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.168947 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:01:05.170380 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.169990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55758)
0929 16:01:05.170151 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.170242 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:01:05.170637 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.238974 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 2
I20260929 16:01:05.240769 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.239977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55774)
0929 16:01:05.240344 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.240518 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 16:01:05.241988 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.241558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55780)
0929 16:01:05.241757 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.241849 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:01:05.242266 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.256126 19211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 3
I20260929 16:01:05.257761 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.257020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55782)
0929 16:01:05.257356 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.257519 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 16:01:05.259145 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.258607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55784)
0929 16:01:05.258893 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.258981 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260929 16:01:05.259408 19211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.348539 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 3
I20260929 16:01:05.350164 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.349518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55788)
0929 16:01:05.349792 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.349935 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 16:01:05.351668 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.351177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55794)
0929 16:01:05.351434 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.351532 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 16:01:05.351904 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.402712 19211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 4
I20260929 16:01:05.404207 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.403573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55800)
0929 16:01:05.403860 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.403965 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:01:05.405438 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.404891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55802)
0929 16:01:05.405214 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.405306 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260929 16:01:05.405678 19211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
W20260929 16:01:05.434211 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:05.434609 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:05.434770 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:05.442888 19219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:05.443032 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:05.443128 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:05.443226 19219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:05.443272 19219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:05.454777 19219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35943,127.5.117.61:32907,127.5.117.60:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.117.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:05.456653 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:05.458282 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:05.471884 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:05.472306 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:05.472301 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:05.473022 19219 server_base.cc:1034] running on GCE node
I20260929 16:01:05.473879 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:05.475546 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:05.476794 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790697665476746 us; error 53 us; skew 500 ppm
I20260929 16:01:05.477226 19219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:05.480849 19219 init.cc:377] Logged in from keytab as kudu/127.5.117.60@KRBTEST.COM (short username kudu)
I20260929 16:01:05.482537 19219 webserver.cc:460] Webserver started at http://127.5.117.60:42861/ using document root <none> and password file <none>
I20260929 16:01:05.483711 19219 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:05.483872 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:05.484488 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:05.488195 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data/instance:
uuid: "0a39931de6d84f4388756495d74155d3"
format_stamp: "Formatted at 2026-09-29 16:01:05 on dist-test-slave-b69v"
server_key: "6ba60fcf2da6ff103452ace37e78b52e"
server_key_iv: "8c4b0ba5fbbe4e75e7f268229ddaef91"
server_key_version: "encryptionkey@0"
I20260929 16:01:05.489109 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal/instance:
uuid: "0a39931de6d84f4388756495d74155d3"
format_stamp: "Formatted at 2026-09-29 16:01:05 on dist-test-slave-b69v"
server_key: "6ba60fcf2da6ff103452ace37e78b52e"
server_key_iv: "8c4b0ba5fbbe4e75e7f268229ddaef91"
server_key_version: "encryptionkey@0"
I20260929 16:01:05.495245 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:01:05.499615 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:05.501543 19219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:05.501745 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "0a39931de6d84f4388756495d74155d3"
format_stamp: "Formatted at 2026-09-29 16:01:05 on dist-test-slave-b69v"
server_key: "6ba60fcf2da6ff103452ace37e78b52e"
server_key_iv: "8c4b0ba5fbbe4e75e7f268229ddaef91"
server_key_version: "encryptionkey@0"
I20260929 16:01:05.501963 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:05.518244 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:05.523149 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:05.523566 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:05.529124 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 4
I20260929 16:01:05.530690 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.530132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55806)
0929 16:01:05.530386 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.530523 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:01:05.532124 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.531635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55816)
0929 16:01:05.531816 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.531931 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:01:05.532404 19137 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:38239: Network error: Client connection negotiation failed: client connection to 127.5.117.60:38239: connect: Connection refused (error 111)
I20260929 16:01:05.547866 19219 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.60:38239
I20260929 16:01:05.547885 19285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.60:38239 every 8 connection(s)
I20260929 16:01:05.549675 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
I20260929 16:01:05.554489 19286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:05.559356 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19219
I20260929 16:01:05.559597 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697441705944-5588-0/minicluster-data/master-2/wal/instance
I20260929 16:01:05.560294 5588 external_mini_cluster.cc:1428] Setting key 418c25e5078cd53a1e7886c954529f04
I20260929 16:01:05.563652 19286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:01:05.586316 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.567451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.5.117.62:35943)
0929 16:01:05.568113 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.568120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.568617 (+ 497us) server_negotiation.cc:407] Connection header received
0929 16:01:05.569016 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.569021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.569128 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.569232 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.571498 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.572390 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.574038 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.574585 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.574707 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.578935 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.578940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.578944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.579014 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.581891 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.582746 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.582755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.582762 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.582908 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.583342 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.583348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.583351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.583646 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.583811 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.585603 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.585843 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":288,"threads_started":1}
I20260929 16:01:05.586860 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.567435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35943 (local address 127.0.0.1:39024)
0929 16:01:05.568431 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.568484 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.568721 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.569337 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.569350 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.570258 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.571276 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.571300 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.572565 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.572571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.573884 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.573895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.574823 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.574830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.575253 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.576150 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.576174 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.578729 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.582035 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.582047 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.582058 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.582621 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.583022 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.583031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.583035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.583225 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.583785 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.583794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.584260 (+ 466us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.584614 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.585168 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":98,"thread_start_us":607,"threads_started":1}
I20260929 16:01:05.586885 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.563310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.5.117.60:38239)
0929 16:01:05.564679 (+ 1369us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.564699 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.565368 (+ 669us) server_negotiation.cc:407] Connection header received
0929 16:01:05.568144 (+ 2776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.568186 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.568726 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.571172 (+ 2446us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.571371 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.572784 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.574340 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.575189 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.575306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.580153 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.580164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.580177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.580207 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.583257 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.583942 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.583951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.583954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.584182 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.584691 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.584696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.584699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.585288 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.585607 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.586033 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.586313 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":1024,"thread_start_us":662,"threads_started":1}
I20260929 16:01:05.588717 19286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:05.604954 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.589454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:32907 (local address 127.0.0.1:55388)
0929 16:01:05.589728 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.589748 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.589842 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.590380 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.590384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.590900 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.591126 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.591138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.592255 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.592259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.593362 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.593376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.594197 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.594203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.594427 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.595087 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.595113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.597737 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.601687 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.601698 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.601705 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.602457 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.603133 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.603138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.603142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.603333 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.603959 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.603965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.604187 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.604369 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.604672 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 16:01:05.605647 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.589609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.5.117.61:32907)
0929 16:01:05.589859 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.589866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.589887 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:01:05.589967 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.589972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.590065 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.590174 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.591305 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.592118 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.593585 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.594049 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.594155 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.597939 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.597944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.597947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.597982 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.601434 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.602687 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.602696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.602708 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.602954 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.603485 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.603490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.603496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.603835 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.603978 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.604568 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.604920 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:01:05.606900 19286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:01:05.607312 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.590061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.5.117.62:35943)
0929 16:01:05.590231 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.590235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.590354 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:01:05.590518 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.590521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.590590 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.590687 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.592073 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.592848 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.594337 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.594882 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.595017 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.600161 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.600166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.600170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.600202 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.603734 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.604638 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.604645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.604649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.604897 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.605457 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.605461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.605463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.605889 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.606201 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.606716 (+ 515us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.606982 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697665, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:01:05.621744 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.607920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55828)
0929 16:01:05.608261 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.608283 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.608432 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.608866 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.608870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.609430 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.609628 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.609637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.610616 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.610624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.611517 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.611527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.612297 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.612302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.612566 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.613165 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.613183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.615256 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.618888 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.618893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.618898 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.619437 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.619812 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.619818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.619821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.619989 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.620742 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.620746 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.620882 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.621023 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.621416 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":41}
I20260929 16:01:05.622004 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.608129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.5.117.60:38239)
0929 16:01:05.608273 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.608280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.608338 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:01:05.608584 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.608590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.608677 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.608788 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.609763 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.610492 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.611660 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.612159 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.612288 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.615464 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.615473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.615478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.615519 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.618751 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.619551 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.619559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.619563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.619694 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.620159 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.620164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.620168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.620613 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.620779 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.621155 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.621597 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":205,"server-negotiator.queue_time_us":62}
I20260929 16:01:05.627229 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.609428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.5.117.61:32907)
0929 16:01:05.609632 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.609637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.609655 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:01:05.609826 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.609832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.609905 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.610004 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.611303 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.612171 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.613428 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.613882 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.614022 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.619400 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.619405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.619408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.619443 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.623876 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.625154 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.625158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.625160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.625314 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.625869 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.625873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.625875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.626161 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.626379 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.626772 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.626970 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:01:05.629850 19286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: Bootstrap starting.
I20260929 16:01:05.632886 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:01:05.634539 19286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:05.636176 19286 log.cc:824] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:05.640093 19286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: No bootstrap required, opened a new log
I20260929 16:01:05.653036 19286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.653890 19286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:05.654145 19286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a39931de6d84f4388756495d74155d3, State: Initialized, Role: FOLLOWER
I20260929 16:01:05.655321 19286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.662724 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:05.664259 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.640287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.5.117.60:38239)
0929 16:01:05.640892 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.640900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.641692 (+ 792us) server_negotiation.cc:407] Connection header received
0929 16:01:05.642029 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.642038 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.642125 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.642206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.644823 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.646061 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.648561 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.649365 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.649465 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.654131 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.654139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.654144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.654188 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.660027 (+ 5839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.661275 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.661284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.661290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.661515 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.662101 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.662109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.662114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.662632 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.662913 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.663599 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.663851 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
I20260929 16:01:05.665436 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:05.666402 19286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:05.673905 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.640417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.5.117.62:35943)
0929 16:01:05.640674 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.640682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.651069 (+ 10387us) server_negotiation.cc:407] Connection header received
0929 16:01:05.651748 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.651758 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.651903 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.651998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.654878 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.656458 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.659732 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.660427 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.660534 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.664238 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.664244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.664250 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.664314 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.670246 (+ 5932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.671156 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.671164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.671168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.671325 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.671806 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.671811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.671814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.672255 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.672534 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.673183 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.673464 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:01:05.676565 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.642099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.5.117.61:32907)
0929 16:01:05.642351 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.642359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.659562 (+ 17203us) server_negotiation.cc:407] Connection header received
0929 16:01:05.659702 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.659708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.659810 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.659901 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.661951 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.663084 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.665245 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.665894 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.666011 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.669413 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.669422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.669429 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.669484 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.673168 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.673953 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.673959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.673963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.674117 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.674742 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.674746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.674749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.675159 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.675426 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.676016 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.676312 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 16:01:05.684758 19311 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:01:05.684916 19311 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:01:05.675630 19219 master_runner.cc:186] Error getting master registration for 127.5.117.62:35943: OK, instance_id { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" instance_seqno: 1790697664380254 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:01:05.687067 19219 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:01:05.704947 19211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 5
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:01:05.720492 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.705709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55842)
0929 16:01:05.705954 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.705973 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.706098 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.706497 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.706500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.707004 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.707282 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.707295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.708787 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.708790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.709651 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.709662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.710657 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.710661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.710814 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.711554 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.711572 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.714019 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.717921 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.717927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.717931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.718324 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.718726 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.718732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.718735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.718870 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.719414 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.719419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.719576 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.719724 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.720043 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:01:05.720486 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.705805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.5.117.60:38239)
0929 16:01:05.706029 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.706058 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.706089 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:01:05.706271 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.706275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.706375 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.706473 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.707436 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.708658 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.709790 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.710513 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.710649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.714211 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.714218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.714222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.714262 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.717779 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.718434 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.718441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.718444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.718596 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.718975 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.718980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.718983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.719295 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.719456 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.719836 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.720116 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:01:05.727869 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: Bootstrap starting.
I20260929 16:01:05.733187 19211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:05.734813 19211 log.cc:824] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:05.738848 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: No bootstrap required, opened a new log
I20260929 16:01:05.750803 19211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.751317 19211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:05.751436 19211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e283348c28543cda9db79036c8a4fe1, State: Initialized, Role: FOLLOWER
I20260929 16:01:05.752255 19211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.753332 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:05.753722 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:05.754980 19211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:01:05.768512 19325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:01:05.768800 19325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:01:05.763262 19144 master_runner.cc:186] Error getting master registration for 127.5.117.62:35943: OK, instance_id { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" instance_seqno: 1790697664380254 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:01:05.775837 19144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:01:05.776393 19240 tablet.cc:2369] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:05.809648 19137 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } attempt: 5
Sep 29 16:01:05 dist-test-slave-b69v krb5kdc[19057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:01:05.830051 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.810368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:38239 (local address 127.0.0.1:55854)
0929 16:01:05.810664 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.810689 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:05.810807 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:05.811450 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:05.811457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:05.812124 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:05.812377 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.812404 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.814061 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.814067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.815645 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:05.815660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.816701 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.816707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:05.816898 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.817729 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:05.817756 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:05.820873 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:05.825116 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.825122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.825127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.825659 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.826065 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:05.826070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:05.826074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:05.826285 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:05.826949 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:05.826954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:05.827121 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 16:01:05.827283 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 16:01:05.827576 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 16:01:05.830255 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.810868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.5.117.60:38239)
0929 16:01:05.811100 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:05.811107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:05.811128 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:01:05.811187 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:05.811193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:05.811276 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:05.811392 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:05.812577 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.813848 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.815810 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:05.816554 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:05.816679 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:05.821095 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:05.821103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:05.821109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:05.821150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:05.824964 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.825789 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.825796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.825800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.825950 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:05.826428 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:05.826434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:05.826438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:05.826820 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:05.826991 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:05.827392 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 16:01:05.829820 (+ 2428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:01:05.837358 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: Bootstrap starting.
I20260929 16:01:05.842067 19137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:05.843557 19137 log.cc:824] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:05.848507 19137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: No bootstrap required, opened a new log
I20260929 16:01:05.854988 19300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:05.855332 19300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.857131 19300 leader_election.cc:288] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19ea96b60c5840778174bd6f4154e4a5 (127.5.117.62:35943), 3e283348c28543cda9db79036c8a4fe1 (127.5.117.61:32907)
I20260929 16:01:05.859145 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a39931de6d84f4388756495d74155d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e283348c28543cda9db79036c8a4fe1" is_pre_election: true
I20260929 16:01:05.859786 19186 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a39931de6d84f4388756495d74155d3 in term 0.
I20260929 16:01:05.859874 19137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.860304 19137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:05.860441 19137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ea96b60c5840778174bd6f4154e4a5, State: Initialized, Role: FOLLOWER
I20260929 16:01:05.860636 19235 leader_election.cc:302] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a39931de6d84f4388756495d74155d3, 3e283348c28543cda9db79036c8a4fe1; no voters:
I20260929 16:01:05.861267 19300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:05.861112 19137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.861428 19300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:05.861538 19300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:05.868413 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:05.870692 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:05.870560 19137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:05.870604 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a39931de6d84f4388756495d74155d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ea96b60c5840778174bd6f4154e4a5" is_pre_election: true
I20260929 16:01:05.871220 19112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a39931de6d84f4388756495d74155d3 in term 0.
I20260929 16:01:05.870169 19300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.875016 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a39931de6d84f4388756495d74155d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19ea96b60c5840778174bd6f4154e4a5"
I20260929 16:01:05.875223 19112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:05.878804 19112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a39931de6d84f4388756495d74155d3 in term 1.
I20260929 16:01:05.879254 19235 leader_election.cc:302] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a39931de6d84f4388756495d74155d3, 19ea96b60c5840778174bd6f4154e4a5; no voters:
I20260929 16:01:05.882558 19300 leader_election.cc:288] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [CANDIDATE]: Term 1 election: Requested vote from peers 19ea96b60c5840778174bd6f4154e4a5 (127.5.117.62:35943), 3e283348c28543cda9db79036c8a4fe1 (127.5.117.61:32907)
I20260929 16:01:05.883054 19300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:05.883566 19300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [term 1 LEADER]: Becoming Leader. State: Replica: 0a39931de6d84f4388756495d74155d3, State: Running, Role: LEADER
I20260929 16:01:05.884446 19300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } }
I20260929 16:01:05.897518 19186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a39931de6d84f4388756495d74155d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e283348c28543cda9db79036c8a4fe1"
I20260929 16:01:05.897801 19186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:05.902331 19186 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a39931de6d84f4388756495d74155d3 in term 1.
I20260929 16:01:05.907588 19331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a39931de6d84f4388756495d74155d3. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:05.908152 19331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:05.912640 19336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:05.915169 19336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:05.935863 19186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [term 1 FOLLOWER]: Refusing update from remote peer 0a39931de6d84f4388756495d74155d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:05.937148 19300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:05.944757 19112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [term 1 FOLLOWER]: Refusing update from remote peer 0a39931de6d84f4388756495d74155d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:05.945885 19331 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:01:05.958598 19165 tablet.cc:2369] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:05.973985 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a39931de6d84f4388756495d74155d3. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:05.974431 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:05.987521 19337 mvcc.cc:205] Tried to move back new op lower bound from 7334697639641714688 to 7334697639476506624. Current Snapshot: MvccSnapshot[applied={T|T < 7334697639641714688}]
I20260929 16:01:06.000819 19300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:06.001273 19300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:06.003520 19328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:06.003841 19328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:06.006175 19336 catalog_manager.cc:1347] Generated new cluster ID: 07c971d82b96456a9f767b1130263e97
I20260929 16:01:06.006295 19336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:06.028721 19347 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:01:06.029233 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a39931de6d84f4388756495d74155d3. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:06.029331 19331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:06.029608 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:06.029691 19331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:06.031816 19340 mvcc.cc:205] Tried to move back new op lower bound from 7334697639641714688 to 7334697639476506624. Current Snapshot: MvccSnapshot[applied={T|T < 7334697639641714688}]
I20260929 16:01:06.042582 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0a39931de6d84f4388756495d74155d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ea96b60c5840778174bd6f4154e4a5" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35943 } } peers { permanent_uuid: "3e283348c28543cda9db79036c8a4fe1" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 32907 } } peers { permanent_uuid: "0a39931de6d84f4388756495d74155d3" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 38239 } } }
I20260929 16:01:06.042845 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:01:06.051249 19336 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:01:06.053393 19336 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:06.074541 19336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3: Generated new TSK 0
I20260929 16:01:06.075920 19336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:01:06.771515 19325 catalog_manager.cc:1259] Loaded cluster ID: 07c971d82b96456a9f767b1130263e97
I20260929 16:01:06.771643 19325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: loading cluster ID for follower catalog manager: success
I20260929 16:01:06.774559 19325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: acquiring CA information for follower catalog manager: success
I20260929 16:01:06.776598 19325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e283348c28543cda9db79036c8a4fe1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:01:07.034665 19347 catalog_manager.cc:1259] Loaded cluster ID: 07c971d82b96456a9f767b1130263e97
I20260929 16:01:07.034806 19347 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: loading cluster ID for follower catalog manager: success
I20260929 16:01:07.038546 19347 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: acquiring CA information for follower catalog manager: success
I20260929 16:01:07.041309 19347 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19ea96b60c5840778174bd6f4154e4a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:01:07.187920 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19070
I20260929 16:01:07.189747 19132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:07.312294 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19070
I20260929 16:01:07.336337 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19144
I20260929 16:01:07.337601 19206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:07.450481 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:07.449546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35943 (local address 127.0.0.1:39060)
0929 16:01:07.450113 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:07.450273 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":304,"threads_started":1}
W20260929 16:01:07.450922 19235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.62:35943: connect: Connection refused (error 111)
I20260929 16:01:07.452476 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:07.452132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35943 (local address 127.0.0.1:39074)
0929 16:01:07.452261 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:07.452345 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:35943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 16:01:07.452725 19235 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a39931de6d84f4388756495d74155d3 -> Peer 19ea96b60c5840778174bd6f4154e4a5 (127.5.117.62:35943): Couldn't send request to peer 19ea96b60c5840778174bd6f4154e4a5. Status: Network error: Client connection negotiation failed: client connection to 127.5.117.62:35943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:01:07.460652 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19144
I20260929 16:01:07.486061 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19219
I20260929 16:01:07.487160 19281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:07.612823 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19219
2026-09-29T16:01:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5183 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:01:07 dist-test-slave-b69v krb5kdc[19362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:01:07 dist-test-slave-b69v krb5kdc[19362](info): set up 2 sockets
Sep 29 16:01:07 dist-test-slave-b69v krb5kdc[19362](info): commencing operation
krb5kdc: starting...
W20260929 16:01:08.769634 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:01:08 dist-test-slave-b69v krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:01:09.021744 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38825
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:09.393443 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:09.393908 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:09.394074 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:09.401899 19375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:09.402065 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:09.402181 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:09.402292 19375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:09.402374 19375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:09.413166 19375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:38825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:09.414781 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:09.416311 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:09.429870 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:09.429968 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:09.429992 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:09.431077 19375 server_base.cc:1034] running on GCE node
I20260929 16:01:09.431959 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:09.433516 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:09.434754 19375 hybrid_clock.cc:648] HybridClock initialized: now 1790697669434695 us; error 45 us; skew 500 ppm
I20260929 16:01:09.435165 19375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:09 dist-test-slave-b69v krb5kdc[19362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:09.444667 19375 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:09.446561 19375 webserver.cc:460] Webserver started at http://127.5.117.62:35823/ using document root <none> and password file <none>
I20260929 16:01:09.447395 19375 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:09.447508 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:09.448009 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:09.452332 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "773ec8537f0648a5998abda687337d97"
format_stamp: "Formatted at 2026-09-29 16:01:09 on dist-test-slave-b69v"
server_key: "74a1a2a8ea7d9123a94d12c8ac52805f"
server_key_iv: "0bcabc3cd32bec1cf93267ab050c409e"
server_key_version: "encryptionkey@0"
I20260929 16:01:09.453308 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "773ec8537f0648a5998abda687337d97"
format_stamp: "Formatted at 2026-09-29 16:01:09 on dist-test-slave-b69v"
server_key: "74a1a2a8ea7d9123a94d12c8ac52805f"
server_key_iv: "0bcabc3cd32bec1cf93267ab050c409e"
server_key_version: "encryptionkey@0"
I20260929 16:01:09.460187 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 16:01:09.464600 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:09.466252 19375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:01:09.466488 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "773ec8537f0648a5998abda687337d97"
format_stamp: "Formatted at 2026-09-29 16:01:09 on dist-test-slave-b69v"
server_key: "74a1a2a8ea7d9123a94d12c8ac52805f"
server_key_iv: "0bcabc3cd32bec1cf93267ab050c409e"
server_key_version: "encryptionkey@0"
I20260929 16:01:09.466749 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:09.486919 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:09.491863 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:09.492237 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:09.514668 19375 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:38825
I20260929 16:01:09.514695 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:38825 every 8 connection(s)
I20260929 16:01:09.516352 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:09.520769 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:09.524477 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19375
I20260929 16:01:09.524735 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:01:09.525638 5588 external_mini_cluster.cc:1428] Setting key 5e8b8882c057bb09836738e28678aa75
I20260929 16:01:09.535013 19442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97: Bootstrap starting.
I20260929 16:01:09.539423 19442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:09.540879 19442 log.cc:824] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:01:09 dist-test-slave-b69v krb5kdc[19362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:09.544445 19442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97: No bootstrap required, opened a new log
I20260929 16:01:09.551276 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:09.527813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.5.117.62:38825)
0929 16:01:09.528683 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:09.528697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:09.528741 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:01:09.530564 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:09.530586 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:09.530988 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:09.531339 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:09.533347 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:09.534780 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:09.536693 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:09.537392 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:09.537538 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:09.543404 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:09.543412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:09.543426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:09.543466 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:09.546882 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:09.547793 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:09.547800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:09.547805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:09.548017 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:09.548505 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:09.548509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:09.548511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:09.549053 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:09.549305 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:09.549806 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 16:01:09.550219 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":451,"threads_started":1}
I20260929 16:01:09.553969 19442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } }
I20260929 16:01:09.554399 19442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:09.554527 19442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773ec8537f0648a5998abda687337d97, State: Initialized, Role: FOLLOWER
I20260929 16:01:09.555145 19442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } }
I20260929 16:01:09.555414 19442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:09.555573 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:01:09.555588 19442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:09.555760 19442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:09.557953 19442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } }
I20260929 16:01:09.558475 19442 leader_election.cc:302] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 773ec8537f0648a5998abda687337d97; no voters:
I20260929 16:01:09.559146 19442 leader_election.cc:288] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:01:09.559237 19447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:09.560036 19447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [term 1 LEADER]: Becoming Leader. State: Replica: 773ec8537f0648a5998abda687337d97, State: Running, Role: LEADER
I20260929 16:01:09.560577 19447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } }
I20260929 16:01:09.561360 19442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:09.563886 19448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "773ec8537f0648a5998abda687337d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } } }
I20260929 16:01:09.563869 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 773ec8537f0648a5998abda687337d97. Latest consensus state: current_term: 1 leader_uuid: "773ec8537f0648a5998abda687337d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "773ec8537f0648a5998abda687337d97" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38825 } } }
I20260929 16:01:09.564375 19448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:09.564689 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:09.565738 19452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:09.571545 19452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:09.583230 19452 catalog_manager.cc:1347] Generated new cluster ID: 113456ea40d54cccbfa94d77eab3a0b0
I20260929 16:01:09.583431 19452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:09.594046 19452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:01:09.595611 19452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:09.605655 19452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 773ec8537f0648a5998abda687337d97: Generated new TSK 0
I20260929 16:01:09.606729 19452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:01:09.617257 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19375
I20260929 16:01:09.618713 19437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:09.713825 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19375
2026-09-29T16:01:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2088 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:01:09 dist-test-slave-b69v krb5kdc[19470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:01:09 dist-test-slave-b69v krb5kdc[19470](info): set up 2 sockets
Sep 29 16:01:09 dist-test-slave-b69v krb5kdc[19470](info): commencing operation
krb5kdc: starting...
W20260929 16:01:10.837118 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:01:11 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:01:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:01:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:01:11.110524 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:11.487922 19483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:11.488751 19483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:11.488972 19483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:11.498221 19483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:11.498384 19483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:11.498529 19483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:11.498622 19483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:11.498716 19483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:11.510459 19483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:11.512274 19483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:11.513893 19483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:11.527029 19488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:11.527004 19489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:11.527091 19491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:11.528360 19483 server_base.cc:1034] running on GCE node
I20260929 16:01:11.529415 19483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:11.530869 19483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:11.532150 19483 hybrid_clock.cc:648] HybridClock initialized: now 1790697671532100 us; error 58 us; skew 500 ppm
I20260929 16:01:11.532531 19483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:11 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:11.536329 19483 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:11.537794 19483 webserver.cc:460] Webserver started at http://127.5.117.62:40345/ using document root <none> and password file <none>
I20260929 16:01:11.538532 19483 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:11.538687 19483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:11.539012 19483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:11.542243 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "37e7517170d543de8e6ce939a0140999"
format_stamp: "Formatted at 2026-09-29 16:01:11 on dist-test-slave-b69v"
server_key: "3b426ece99f250a98c08372e67a23206"
server_key_iv: "53d9af8043a919a5633a23b6b7375017"
server_key_version: "encryptionkey@0"
I20260929 16:01:11.543073 19483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "37e7517170d543de8e6ce939a0140999"
format_stamp: "Formatted at 2026-09-29 16:01:11 on dist-test-slave-b69v"
server_key: "3b426ece99f250a98c08372e67a23206"
server_key_iv: "53d9af8043a919a5633a23b6b7375017"
server_key_version: "encryptionkey@0"
I20260929 16:01:11.548576 19483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:01:11.552176 19498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:11.553675 19483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:01:11.553886 19483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "37e7517170d543de8e6ce939a0140999"
format_stamp: "Formatted at 2026-09-29 16:01:11 on dist-test-slave-b69v"
server_key: "3b426ece99f250a98c08372e67a23206"
server_key_iv: "53d9af8043a919a5633a23b6b7375017"
server_key_version: "encryptionkey@0"
I20260929 16:01:11.554105 19483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:11.575730 19483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:11.581494 19483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:11.581864 19483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:11.606403 19483 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46489
I20260929 16:01:11.606431 19549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46489 every 8 connection(s)
I20260929 16:01:11.608210 19483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:11.610105 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19483
I20260929 16:01:11.610356 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:01:11.611330 5588 external_mini_cluster.cc:1428] Setting key 116844e4b3d87a83a6221d044d88182c
I20260929 16:01:11.614018 19550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:01:11 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:11.630359 19550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap starting.
I20260929 16:01:11.635593 19550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:11.637245 19550 log.cc:824] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:11.641389 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:11.613850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.5.117.62:46489)
0929 16:01:11.614765 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:11.614780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:11.614824 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:01:11.616801 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:11.616826 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:11.617256 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:11.617645 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:11.619959 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.621525 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.623512 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.624424 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.624574 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:11.630396 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:11.630413 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:11.630438 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:11.630502 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:11.635106 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.636360 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.636371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.636380 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.636802 (+ 422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.637551 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.637556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.637558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.638476 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:11.638871 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:11.639320 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:01:11.639835 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":451,"threads_started":1}
I20260929 16:01:11.641618 19550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: No bootstrap required, opened a new log
I20260929 16:01:11.650604 19550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:11.651096 19550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:11.651228 19550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Initialized, Role: FOLLOWER
I20260929 16:01:11.651899 19550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:11.652263 19550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:11.652463 19550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:11.652679 19550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:11.655254 19550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:11.655967 19550 leader_election.cc:302] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e7517170d543de8e6ce939a0140999; no voters:
I20260929 16:01:11.656804 19550 leader_election.cc:288] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:01:11.656919 19555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:11.657857 19555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 LEADER]: Becoming Leader. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Running, Role: LEADER
I20260929 16:01:11.658506 19555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:11.659544 19550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:11.661600 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:11.662792 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:11.663986 19557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e7517170d543de8e6ce939a0140999. Latest consensus state: current_term: 1 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:11.664716 19557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:11.665776 19562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:11.672252 19562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:11.683295 19562 catalog_manager.cc:1347] Generated new cluster ID: 2c470e4e6c0c4a5f8213dfe808f813f6
I20260929 16:01:11.683461 19562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:11.694705 19562 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:01:11.696966 19562 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:11.707360 19562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Generated new TSK 0
I20260929 16:01:11.708709 19562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:01:11.795858 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:12.188431 19576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:12.188769 19576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:12.188948 19576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:12.196238 19576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:12.196384 19576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:12.196571 19576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:01:12.206321 19576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:12.207834 19576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:12.209375 19576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:12.224424 19582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:12.224514 19584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:12.224459 19581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:12.225368 19576 server_base.cc:1034] running on GCE node
I20260929 16:01:12.226073 19576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:12.227265 19576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:12.228490 19576 hybrid_clock.cc:648] HybridClock initialized: now 1790697672228425 us; error 57 us; skew 500 ppm
I20260929 16:01:12.228838 19576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:12 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:12.232496 19576 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:01:12.234246 19576 webserver.cc:460] Webserver started at http://127.5.117.1:42877/ using document root <none> and password file <none>
I20260929 16:01:12.235056 19576 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:12.235216 19576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:12.235561 19576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:12.238866 19576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "80e1d8d4073a418baaa23fac399b6812"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "0787c0ca4239cffbd81f7bcb09c1ab5c"
server_key_iv: "ecbdd27ab0e475054c0dbbd30564a833"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.239693 19576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "80e1d8d4073a418baaa23fac399b6812"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "0787c0ca4239cffbd81f7bcb09c1ab5c"
server_key_iv: "ecbdd27ab0e475054c0dbbd30564a833"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.245869 19576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 16:01:12.250180 19591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.251537 19576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:01:12.251717 19576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "80e1d8d4073a418baaa23fac399b6812"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "0787c0ca4239cffbd81f7bcb09c1ab5c"
server_key_iv: "ecbdd27ab0e475054c0dbbd30564a833"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.251920 19576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:12.287447 19576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:12.292349 19576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:12.292698 19576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:12.293787 19576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:12.295270 19576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:12.295378 19576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.295540 19576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:12.295624 19576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.333971 19576 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36765
I20260929 16:01:12.334017 19703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36765 every 8 connection(s)
I20260929 16:01:12.335558 19576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:12.339610 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19576
I20260929 16:01:12.339864 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:01:12.340724 5588 external_mini_cluster.cc:1428] Setting key 2dadeae06813e5d1f23551e123eb8176
Sep 29 16:01:12 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:12.362543 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:12.338760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:58371 (local address 127.5.117.62:46489)
0929 16:01:12.339564 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:12.339574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:12.341403 (+ 1829us) server_negotiation.cc:407] Connection header received
0929 16:01:12.343213 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:12.343220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:12.343313 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:12.343435 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:12.345829 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.346647 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.349312 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.349734 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.349840 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:12.354435 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:12.354443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:12.354452 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:12.354495 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:12.358183 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:12.359192 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:12.359201 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:12.359205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:12.359459 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:12.360215 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:12.360223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:12.360228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:12.360599 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:12.360811 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:12.361666 (+ 855us) server_negotiation.cc:299] Negotiation successful
0929 16:01:12.361989 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":388,"threads_started":1}
I20260929 16:01:12.363739 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:12.339604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:58371)
0929 16:01:12.340628 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:12.340687 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:12.342885 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:12.343752 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:12.343764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:12.344674 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:12.345617 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:12.345647 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.346821 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.346831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:12.349128 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:12.349149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.349987 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.349998 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:12.350365 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:12.351229 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:12.351270 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:12.354188 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:12.358318 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:12.358324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:12.358335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:12.358852 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:12.359695 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:12.359701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:12.359704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:12.360006 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:12.360770 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:12.360782 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:12.361204 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 16:01:12.361517 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 16:01:12.362352 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":464,"threads_started":1}
I20260929 16:01:12.365756 19704 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:12.366130 19704 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:12.367069 19704 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:12.369668 19515 ts_manager.cc:194] Registered new tserver with Master: 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765)
I20260929 16:01:12.371295 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:58371
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:01:12.410099 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:12.785244 19710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:12.785548 19710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:12.785645 19710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:12.792994 19710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:12.793099 19710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:12.793212 19710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:01:12.802729 19710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:12.804105 19710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:12.805524 19710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:12.819573 19716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:12.820120 19718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:12.820354 19715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:12.820782 19710 server_base.cc:1034] running on GCE node
I20260929 16:01:12.821516 19710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:12.822424 19710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:12.823627 19710 hybrid_clock.cc:648] HybridClock initialized: now 1790697672823583 us; error 31 us; skew 500 ppm
I20260929 16:01:12.823916 19710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:12 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:12.827214 19710 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:01:12.828811 19710 webserver.cc:460] Webserver started at http://127.5.117.2:42765/ using document root <none> and password file <none>
I20260929 16:01:12.829558 19710 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:12.829691 19710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:12.830024 19710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:12.833212 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "f5c9344cbdbdac39f1695f971d365707"
server_key_iv: "b1a32c95d4dd489ac29ff4a9f5eee70e"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.834121 19710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "f5c9344cbdbdac39f1695f971d365707"
server_key_iv: "b1a32c95d4dd489ac29ff4a9f5eee70e"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.839612 19710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:01:12.843111 19725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.844484 19710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:12.844641 19710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "f5c9344cbdbdac39f1695f971d365707"
server_key_iv: "b1a32c95d4dd489ac29ff4a9f5eee70e"
server_key_version: "encryptionkey@0"
I20260929 16:01:12.844830 19710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:12.864823 19710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:12.868780 19710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:12.869059 19710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:12.869941 19710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:12.871266 19710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:12.871374 19710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.871542 19710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:12.871618 19710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:12.906209 19710 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44131
I20260929 16:01:12.906287 19837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44131 every 8 connection(s)
I20260929 16:01:12.907500 19710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:12.917943 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19710
I20260929 16:01:12.918181 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:01:12.918888 5588 external_mini_cluster.cc:1428] Setting key dfe31e6697978613db4375bd371c7d2d
Sep 29 16:01:12 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:12.931937 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:12.910054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:39517 (local address 127.5.117.62:46489)
0929 16:01:12.910579 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:12.910586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:12.911500 (+ 914us) server_negotiation.cc:407] Connection header received
0929 16:01:12.913429 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:12.913434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:12.913510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:12.913620 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:12.915200 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.916321 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.918427 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.919180 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.919324 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:12.924729 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:12.924734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:12.924738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:12.924773 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:12.927774 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:12.928894 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:12.928899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:12.928901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:12.929044 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:12.929635 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:12.929639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:12.929641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:12.929958 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:12.930082 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:12.931304 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0929 16:01:12.931598 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260929 16:01:12.933703 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:12.910538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:39517)
0929 16:01:12.911330 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:12.911376 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:12.913171 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:12.913816 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:12.913823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:12.914298 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:12.914865 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:12.914888 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.916539 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.916544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:12.918212 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:12.918231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:12.919582 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:12.919592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:12.919936 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:12.920941 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:12.920978 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:12.924423 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:12.927924 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:12.927937 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:12.927961 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:12.928718 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:12.929177 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:12.929186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:12.929192 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:12.929478 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:12.930093 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:12.930105 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:12.930640 (+ 535us) client_negotiation.cc:769] Sending connection context
0929 16:01:12.931003 (+ 363us) client_negotiation.cc:240] Negotiation successful
0929 16:01:12.931715 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":291,"threads_started":1}
I20260929 16:01:12.935751 19838 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:12.936167 19838 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:12.936985 19838 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:12.938740 19515 ts_manager.cc:194] Registered new tserver with Master: d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131)
I20260929 16:01:12.939693 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:39517
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:01:12.985028 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:13.350217 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:13.350554 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:13.350747 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:13.358444 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:13.358623 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:13.358836 19844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:01:13.369707 19844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:13.371415 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:13.373173 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:01:13.374539 19704 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
W20260929 16:01:13.388710 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.388670 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.388638 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:13.389762 19844 server_base.cc:1034] running on GCE node
I20260929 16:01:13.390419 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:13.391440 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:13.392692 19844 hybrid_clock.cc:648] HybridClock initialized: now 1790697673392648 us; error 37 us; skew 500 ppm
I20260929 16:01:13.393071 19844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:13 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697673, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:13.396847 19844 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:01:13.398386 19844 webserver.cc:460] Webserver started at http://127.5.117.3:33773/ using document root <none> and password file <none>
I20260929 16:01:13.399133 19844 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:13.399281 19844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:13.399611 19844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:13.402773 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "baea97ad1ca643c49d92e959cbfdc568"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-b69v"
server_key: "cfe4c4a578d4281cee9ec239459e741d"
server_key_iv: "f07a1409c5b028169672a92f0517df09"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.403561 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "baea97ad1ca643c49d92e959cbfdc568"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-b69v"
server_key: "cfe4c4a578d4281cee9ec239459e741d"
server_key_iv: "f07a1409c5b028169672a92f0517df09"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.408831 19844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:01:13.412376 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.413749 19844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:01:13.413914 19844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "baea97ad1ca643c49d92e959cbfdc568"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-b69v"
server_key: "cfe4c4a578d4281cee9ec239459e741d"
server_key_iv: "f07a1409c5b028169672a92f0517df09"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.414089 19844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:13.439134 19844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:13.443727 19844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:13.444025 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:13.445151 19844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:13.446723 19844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:13.446803 19844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.446935 19844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:13.446996 19844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.485618 19844 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34939
I20260929 16:01:13.485808 19971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34939 every 8 connection(s)
I20260929 16:01:13.487020 19844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:13.493985 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 19844
I20260929 16:01:13.494208 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:01:13.494731 5588 external_mini_cluster.cc:1428] Setting key e5ceee8f52fe0236c4b4e8136fb45e37
Sep 29 16:01:13 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697673, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:13.510169 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.489874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:35567 (local address 127.5.117.62:46489)
0929 16:01:13.490392 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.490397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.491312 (+ 915us) server_negotiation.cc:407] Connection header received
0929 16:01:13.493424 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.493431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.493526 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.493665 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:13.495567 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.496681 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.498799 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.499225 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.499333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.503135 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:13.503142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:13.503146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:13.503212 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:13.506302 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.507349 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.507354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.507356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.507501 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.507990 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.507995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.507997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.508464 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:13.508664 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.509399 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.509708 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260929 16:01:13.511044 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.490337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:35567)
0929 16:01:13.491138 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.491179 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.493129 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.493918 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.493927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.494554 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:13.495221 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.495380 (+ 159us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.496860 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.496869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.498653 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.498665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.499414 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.499419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.499613 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.500449 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:13.500472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:13.502772 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:13.506557 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:13.506568 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:13.506585 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:13.507188 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:13.507641 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:13.507645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:13.507648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:13.507829 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:13.508631 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:13.508637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:13.508943 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.509244 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.509756 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":347,"threads_started":1}
I20260929 16:01:13.512396 19972 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:13.512703 19972 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:13.513413 19972 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:13.514915 19515 ts_manager.cc:194] Registered new tserver with Master: baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:13.515852 19515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:35567
I20260929 16:01:13.522854 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:13.539954 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.527905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.5.117.62:46489)
0929 16:01:13.528126 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.528136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.528625 (+ 489us) server_negotiation.cc:407] Connection header received
0929 16:01:13.528740 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.528743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.528800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.528877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:13.530248 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.531009 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.532414 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.532787 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.532889 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.535102 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:13.535107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:13.535110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:13.535138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:13.537813 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.538390 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.538395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.538397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.538509 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.538812 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.538815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.538817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.539124 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:13.539316 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.539499 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.539641 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:01:13.543540 19515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:13.551535 19515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:13.593411 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.568558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34939 (local address 127.0.0.1:50454)
0929 16:01:13.584240 (+ 15682us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.584254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.584404 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.585411 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.585416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.585441 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:13.585847 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.585859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.587791 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.587798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.590191 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.590205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.592227 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.592237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.592505 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.592559 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.592684 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.592916 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15591,"mutex_wait_us":1,"thread_start_us":14618,"threads_started":1}
I20260929 16:01:13.593868 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.568471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.5.117.1:36765)
0929 16:01:13.583627 (+ 15156us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.583635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.583667 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:01:13.583789 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.583798 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.584122 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.584321 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:13.585211 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.586951 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.590065 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.591627 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.591755 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.592022 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.593243 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.593444 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14962,"thread_start_us":15473,"threads_started":1}
I20260929 16:01:13.594089 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.568857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50454 (local address 127.5.117.3:34939)
0929 16:01:13.583165 (+ 14308us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.583175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.584389 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 16:01:13.584620 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.584627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.584867 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.585088 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:13.586055 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.587627 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.590359 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.591738 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.591842 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.593465 (+ 1623us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.593542 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.593715 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14120,"thread_start_us":13909,"threads_started":1}
I20260929 16:01:13.594102 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.568079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39340)
0929 16:01:13.581679 (+ 13600us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.581747 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.582164 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.584376 (+ 2212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.584385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.584423 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:13.584969 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.584990 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.587120 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.587126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.589883 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.589897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.592877 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.592884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.593046 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.593086 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.593179 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.593298 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13239,"thread_start_us":16355,"threads_started":1}
I20260929 16:01:13.598639 19639 tablet_service.cc:1467] Processing CreateTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 (DEFAULT_TABLE table=test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:13.598639 19907 tablet_service.cc:1467] Processing CreateTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 (DEFAULT_TABLE table=test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:13.600364 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754d6b20c86247ed83dd48f9df0f28f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:13.600364 19639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754d6b20c86247ed83dd48f9df0f28f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:13.600487 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.584873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:44131 (local address 127.0.0.1:33958)
0929 16:01:13.592585 (+ 7712us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.592603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.592715 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.593863 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.593868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.593896 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:13.595152 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.595164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.596832 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.596840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.598135 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.598147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.599591 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.599598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.599752 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.599775 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.599891 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.600017 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7594,"thread_start_us":7438,"threads_started":1}
I20260929 16:01:13.600574 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.585212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.5.117.2:44131)
0929 16:01:13.593004 (+ 7792us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.593011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.593035 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:01:13.593124 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.593129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.593374 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.593577 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:13.595377 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.596689 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.598296 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.599284 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.599425 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.599630 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.599928 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.600133 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7626,"thread_start_us":7469,"threads_started":1}
I20260929 16:01:13.603991 19773 tablet_service.cc:1467] Processing CreateTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 (DEFAULT_TABLE table=test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:13.605486 19773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754d6b20c86247ed83dd48f9df0f28f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:13.626101 19991 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Bootstrap starting.
I20260929 16:01:13.633304 19991 tablet_bootstrap.cc:654] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:13.634361 19992 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Bootstrap starting.
I20260929 16:01:13.634794 19991 log.cc:824] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:13.638870 19992 tablet_bootstrap.cc:654] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:13.639751 19993 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Bootstrap starting.
I20260929 16:01:13.640370 19992 log.cc:824] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:13.643096 19991 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: No bootstrap required, opened a new log
I20260929 16:01:13.643252 19993 tablet_bootstrap.cc:654] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:13.643399 19991 ts_tablet_manager.cc:1397] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260929 16:01:13.643429 19992 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: No bootstrap required, opened a new log
I20260929 16:01:13.643700 19992 ts_tablet_manager.cc:1397] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 16:01:13.644512 19993 log.cc:824] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:13.647353 19993 tablet_bootstrap.cc:492] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: No bootstrap required, opened a new log
I20260929 16:01:13.647609 19993 ts_tablet_manager.cc:1397] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 16:01:13.651877 19991 raft_consensus.cc:348] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.652297 19991 raft_consensus.cc:374] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:13.652462 19991 raft_consensus.cc:729] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baea97ad1ca643c49d92e959cbfdc568, State: Initialized, Role: FOLLOWER
I20260929 16:01:13.653111 19991 consensus_queue.cc:260] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.654369 19972 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
I20260929 16:01:13.654194 19992 raft_consensus.cc:348] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.654567 19992 raft_consensus.cc:374] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:13.654649 19992 raft_consensus.cc:729] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d00aa67b3f4fa9a5c5ed30422780ec, State: Initialized, Role: FOLLOWER
I20260929 16:01:13.654980 19991 ts_tablet_manager.cc:1428] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20260929 16:01:13.655166 19992 consensus_queue.cc:260] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.656934 19992 ts_tablet_manager.cc:1428] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260929 16:01:13.657438 19838 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
I20260929 16:01:13.658159 19993 raft_consensus.cc:348] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.658603 19993 raft_consensus.cc:374] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:13.658740 19993 raft_consensus.cc:729] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e1d8d4073a418baaa23fac399b6812, State: Initialized, Role: FOLLOWER
I20260929 16:01:13.659337 19993 consensus_queue.cc:260] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
W20260929 16:01:13.659518 19839 tablet.cc:2369] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:13.661137 19993 ts_tablet_manager.cc:1428] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
W20260929 16:01:13.738828 19973 tablet.cc:2369] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:13.838740 19705 tablet.cc:2369] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:13.896554 19998 raft_consensus.cc:482] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:13.896957 19998 raft_consensus.cc:504] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.898953 19998 leader_election.cc:288] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765), baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:13.909870 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.899507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:40739 (local address 127.5.117.3:34939)
0929 16:01:13.899766 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.899774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.900800 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 16:01:13.900939 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.900945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.901048 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.901221 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:13.902497 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.903734 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.907714 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.908648 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.908809 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.908947 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.909394 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.909580 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:01:13.910049 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.899587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:38735 (local address 127.5.117.1:36765)
0929 16:01:13.899821 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.899829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.900799 (+ 970us) server_negotiation.cc:407] Connection header received
0929 16:01:13.900939 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.900945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.901048 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.901221 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:13.902497 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.903747 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.907693 (+ 3946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.908979 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.909122 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.909230 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.909624 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.909777 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:01:13.910619 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.899317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.5.117.2:38735)
0929 16:01:13.900494 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.900526 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.900724 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.901369 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.901378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.901442 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:13.902291 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.902303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.903933 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.903939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.907491 (+ 3552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.907501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.909231 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.909236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.909439 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.909470 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.909559 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.909714 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":286,"thread_start_us":812,"threads_started":1}
I20260929 16:01:13.910761 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.899394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34939 (local address 127.5.117.2:40739)
0929 16:01:13.900613 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.900629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.900731 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.901369 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.901378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.901551 (+ 173us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:13.902191 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.902218 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.903934 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.903939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.907479 (+ 3540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.907501 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.908918 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.908928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.909165 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.909216 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.910103 (+ 887us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.910260 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":331,"spinlock_wait_cycles":30592,"thread_start_us":593,"threads_started":1}
I20260929 16:01:13.912127 19927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754d6b20c86247ed83dd48f9df0f28f2" candidate_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baea97ad1ca643c49d92e959cbfdc568" is_pre_election: true
I20260929 16:01:13.912101 19659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754d6b20c86247ed83dd48f9df0f28f2" candidate_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e1d8d4073a418baaa23fac399b6812" is_pre_election: true
I20260929 16:01:13.912734 19927 raft_consensus.cc:2454] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d00aa67b3f4fa9a5c5ed30422780ec in term 0.
I20260929 16:01:13.912734 19659 raft_consensus.cc:2454] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6d00aa67b3f4fa9a5c5ed30422780ec in term 0.
I20260929 16:01:13.913554 19728 leader_election.cc:302] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80e1d8d4073a418baaa23fac399b6812, baea97ad1ca643c49d92e959cbfdc568, d6d00aa67b3f4fa9a5c5ed30422780ec; no voters:
I20260929 16:01:13.914064 19998 raft_consensus.cc:2781] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:13.914244 19998 raft_consensus.cc:482] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:13.914374 19998 raft_consensus.cc:3037] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:13.917133 19998 raft_consensus.cc:504] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.917930 19998 leader_election.cc:288] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [CANDIDATE]: Term 1 election: Requested vote from peers 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765), baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:13.918483 19659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754d6b20c86247ed83dd48f9df0f28f2" candidate_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80e1d8d4073a418baaa23fac399b6812"
I20260929 16:01:13.918545 19927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754d6b20c86247ed83dd48f9df0f28f2" candidate_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baea97ad1ca643c49d92e959cbfdc568"
I20260929 16:01:13.918730 19659 raft_consensus.cc:3037] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:13.918821 19927 raft_consensus.cc:3037] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:13.921355 19659 raft_consensus.cc:2454] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d00aa67b3f4fa9a5c5ed30422780ec in term 1.
I20260929 16:01:13.921820 19727 leader_election.cc:302] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e1d8d4073a418baaa23fac399b6812, d6d00aa67b3f4fa9a5c5ed30422780ec; no voters:
I20260929 16:01:13.922166 19998 raft_consensus.cc:2781] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:13.922313 19927 raft_consensus.cc:2454] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6d00aa67b3f4fa9a5c5ed30422780ec in term 1.
I20260929 16:01:13.923090 19998 raft_consensus.cc:686] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 LEADER]: Becoming Leader. State: Replica: d6d00aa67b3f4fa9a5c5ed30422780ec, State: Running, Role: LEADER
I20260929 16:01:13.923686 19998 consensus_queue.cc:237] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } }
I20260929 16:01:13.929327 19514 catalog_manager.cc:5495] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec reported cstate change: term changed from 0 to 1, leader changed from <none> to d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2). New cstate: current_term: 1 leader_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } health_report { overall_health: HEALTHY } } }
I20260929 16:01:13.951073 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.945261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.5.117.2:44131)
0929 16:01:13.945548 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.945553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.945569 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:01:13.945663 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.945666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.945750 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.945922 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:13.946340 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.947109 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.948655 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.949143 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.949279 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.949812 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:13.950435 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:13.950616 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.950671 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.950762 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:01:13.961362 19927 raft_consensus.cc:1261] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Refusing update from remote peer d6d00aa67b3f4fa9a5c5ed30422780ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:13.961699 19659 raft_consensus.cc:1261] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Refusing update from remote peer d6d00aa67b3f4fa9a5c5ed30422780ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:13.962785 20002 consensus_queue.cc:1035] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:13.963272 19998 consensus_queue.cc:1035] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:13.979233 20006 mvcc.cc:205] Tried to move back new op lower bound from 7334697672533123072 to 7334697672396722176. Current Snapshot: MvccSnapshot[applied={T|T < 7334697672533123072}]
I20260929 16:01:13.979820 20005 mvcc.cc:205] Tried to move back new op lower bound from 7334697672533123072 to 7334697672396722176. Current Snapshot: MvccSnapshot[applied={T|T < 7334697672533123072}]
I20260929 16:01:13.998075 19514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:13.998399 19514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:14.002808 19514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Sending DeleteTablet for 3 replicas of tablet 754d6b20c86247ed83dd48f9df0f28f2
I20260929 16:01:14.004498 19907 tablet_service.cc:1514] Processing DeleteTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:13 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:50454
I20260929 16:01:14.004586 19639 tablet_service.cc:1514] Processing DeleteTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:13 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:39340
I20260929 16:01:14.004758 19773 tablet_service.cc:1514] Processing DeleteTablet for tablet 754d6b20c86247ed83dd48f9df0f28f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:13 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:33958
I20260929 16:01:14.004966 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19576
I20260929 16:01:14.017539 20017 tablet_replica.cc:330] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: stopping tablet replica
I20260929 16:01:14.017941 20018 tablet_replica.cc:330] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: stopping tablet replica
I20260929 16:01:14.018254 20016 tablet_replica.cc:330] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: stopping tablet replica
I20260929 16:01:14.018421 20017 raft_consensus.cc:2229] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:14.018672 20016 raft_consensus.cc:2229] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:14.019163 20017 raft_consensus.cc:2258] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:14.019171 20016 raft_consensus.cc:2258] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:14.019654 20018 raft_consensus.cc:2229] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:14.020574 20018 raft_consensus.cc:2258] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:14.021708 20016 ts_tablet_manager.cc:1905] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:14.023152 20017 ts_tablet_manager.cc:1905] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:14.024582 20018 ts_tablet_manager.cc:1905] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:14.029393 20016 ts_tablet_manager.cc:1918] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:14.029592 20016 log.cc:1196] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/754d6b20c86247ed83dd48f9df0f28f2
I20260929 16:01:14.030135 20016 ts_tablet_manager.cc:1939] T 754d6b20c86247ed83dd48f9df0f28f2 P baea97ad1ca643c49d92e959cbfdc568: Deleting consensus metadata
Sep 29 16:01:14 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:01:14.032294 19501 catalog_manager.cc:4841] TS baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939): tablet 754d6b20c86247ed83dd48f9df0f28f2 (table test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]) successfully deleted
I20260929 16:01:14.036082 20018 ts_tablet_manager.cc:1918] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:14.036268 20018 log.cc:1196] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/754d6b20c86247ed83dd48f9df0f28f2
I20260929 16:01:14.036934 20018 ts_tablet_manager.cc:1939] T 754d6b20c86247ed83dd48f9df0f28f2 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting consensus metadata
I20260929 16:01:14.038079 20017 ts_tablet_manager.cc:1918] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:14.038316 20017 log.cc:1196] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/754d6b20c86247ed83dd48f9df0f28f2
I20260929 16:01:14.038753 19500 catalog_manager.cc:4841] TS d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131): tablet 754d6b20c86247ed83dd48f9df0f28f2 (table test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]) successfully deleted
I20260929 16:01:14.039050 20017 ts_tablet_manager.cc:1939] T 754d6b20c86247ed83dd48f9df0f28f2 P 80e1d8d4073a418baaa23fac399b6812: Deleting consensus metadata
I20260929 16:01:14.040716 19500 catalog_manager.cc:4841] TS 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765): tablet 754d6b20c86247ed83dd48f9df0f28f2 (table test-table [id=9ca9e033c96d4c1eb6277e17b347ec53]) successfully deleted
I20260929 16:01:14.041152 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:14.006500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.5.117.1:36765)
0929 16:01:14.006758 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:14.006768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:14.018897 (+ 12129us) server_negotiation.cc:407] Connection header received
0929 16:01:14.019221 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:14.019228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:14.019348 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:14.019454 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:14.022232 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.023413 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.025749 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.026418 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.026517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:14.032793 (+ 6276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:14.032808 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:14.032830 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:14.032869 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:14.037409 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.038395 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.038416 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.038419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.038633 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.039209 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.039215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.039218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.039738 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:14.040134 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:14.040491 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 16:01:14.040749 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:01:14.042217 19699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:14.144027 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19576
I20260929 16:01:14.166833 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19710
Sep 29 16:01:14 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:01:14.192822 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:14.167792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.5.117.2:44131)
0929 16:01:14.167999 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:14.168005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:14.168127 (+ 122us) server_negotiation.cc:407] Connection header received
0929 16:01:14.168520 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:14.168527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:14.168633 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:14.168754 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:14.170484 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.171797 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.173341 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.174056 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.174192 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:14.184248 (+ 10056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:14.184277 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:14.184306 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:14.184380 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:14.189203 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.190192 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.190210 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.190213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.190435 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.190846 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.190853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.190857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.191438 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:14.191843 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:14.192028 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:01:14.192336 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:01:14.193822 19833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:14.320256 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19710
I20260929 16:01:14.341895 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19844
Sep 29 16:01:14 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:01:14.358949 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:14.342772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.5.117.3:34939)
0929 16:01:14.342972 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:14.342977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:14.343050 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:01:14.343248 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:14.343252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:14.343331 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:14.343426 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:14.344948 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.345773 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.347081 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.347504 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.347614 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:14.352332 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:14.352350 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:14.352371 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:14.352431 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:14.355626 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.356567 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.356577 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.356579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.356740 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.357128 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.357131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.357134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.357605 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:14.357872 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:14.358319 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 16:01:14.358580 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:01:14.359766 19967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:14.455400 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19844
I20260929 16:01:14.475207 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 19483
I20260929 16:01:14.476665 19545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:14.599454 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 19483
I20260929 16:01:14.626602 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--webserver_interface=127.5.117.62
--webserver_port=40345
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:15.004993 20024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:15.005316 20024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:15.005498 20024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:15.012995 20024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:15.013101 20024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:15.013204 20024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:15.013319 20024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:15.013412 20024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:15.023307 20024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=40345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:15.024761 20024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:15.026368 20024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:15.054229 20030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:15.054283 20033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:15.054320 20031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:15.055462 20024 server_base.cc:1034] running on GCE node
I20260929 16:01:15.056553 20024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:15.058418 20024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:15.059667 20024 hybrid_clock.cc:648] HybridClock initialized: now 1790697675059609 us; error 44 us; skew 500 ppm
I20260929 16:01:15.060248 20024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:15 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:15.068740 20024 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:15.070775 20024 webserver.cc:460] Webserver started at http://127.5.117.62:40345/ using document root <none> and password file <none>
I20260929 16:01:15.071694 20024 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:15.071827 20024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:15.078801 20024 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 16:01:15.082116 20040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:15.083635 20024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:01:15.083928 20024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "37e7517170d543de8e6ce939a0140999"
format_stamp: "Formatted at 2026-09-29 16:01:11 on dist-test-slave-b69v"
server_key: "3b426ece99f250a98c08372e67a23206"
server_key_iv: "53d9af8043a919a5633a23b6b7375017"
server_key_version: "encryptionkey@0"
I20260929 16:01:15.085208 20024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:15.101842 20024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:15.106235 20024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:15.106536 20024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:15.134487 20024 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46489
I20260929 16:01:15.134455 20091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46489 every 8 connection(s)
I20260929 16:01:15.136579 20024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:15.144814 20092 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:01:15.145524 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20024
I20260929 16:01:15.146442 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:36765
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=42877
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:15.149179 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap starting.
I20260929 16:01:15.170557 20092 log.cc:824] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:15.183833 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:01:15.184477 20092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap complete.
I20260929 16:01:15.195101 20092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:15.195973 20092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Initialized, Role: FOLLOWER
I20260929 16:01:15.196703 20092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:15.196983 20092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:15.197129 20092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:15.197319 20092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:01:15.201297 20092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:15.201849 20092 leader_election.cc:302] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e7517170d543de8e6ce939a0140999; no voters:
I20260929 16:01:15.202580 20092 leader_election.cc:288] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:01:15.202709 20096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:01:15.204105 20096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 LEADER]: Becoming Leader. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Running, Role: LEADER
I20260929 16:01:15.204722 20096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:15.205158 20092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:15.207430 20097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:15.207876 20097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:15.208280 20098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e7517170d543de8e6ce939a0140999. Latest consensus state: current_term: 2 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:15.208608 20098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:15.216699 20102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:15.224963 20102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:15.226399 20102 catalog_manager.cc:1259] Loaded cluster ID: 2c470e4e6c0c4a5f8213dfe808f813f6
I20260929 16:01:15.226500 20102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:15.229763 20102 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:01:15.247025 20102 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:01:15.255638 20102 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:15.257591 20102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Loaded TSK: 0
I20260929 16:01:15.258980 20102 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:01:15.533227 20094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:15.533510 20094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:15.533671 20094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:15.540517 20094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:15.540627 20094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:15.540795 20094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:01:15.549612 20094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:36765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=42877
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:15.551010 20094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:15.552567 20094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:15.567590 20120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:15.567675 20119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:15.568868 20122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:15.569162 20094 server_base.cc:1034] running on GCE node
I20260929 16:01:15.569769 20094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:15.570729 20094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:15.571959 20094 hybrid_clock.cc:648] HybridClock initialized: now 1790697675571899 us; error 37 us; skew 500 ppm
I20260929 16:01:15.572295 20094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:15 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:15.576314 20094 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:01:15.577944 20094 webserver.cc:460] Webserver started at http://127.5.117.1:42877/ using document root <none> and password file <none>
I20260929 16:01:15.578655 20094 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:15.578812 20094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:15.586807 20094 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:01:15.590372 20129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:15.592186 20094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:15.592398 20094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "80e1d8d4073a418baaa23fac399b6812"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "0787c0ca4239cffbd81f7bcb09c1ab5c"
server_key_iv: "ecbdd27ab0e475054c0dbbd30564a833"
server_key_version: "encryptionkey@0"
I20260929 16:01:15.593608 20094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:15.621037 20094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:15.626526 20094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:15.626901 20094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:15.628115 20094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:15.629724 20094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:15.629869 20094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:15.630004 20094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:15.630098 20094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:15.668676 20094 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36765
I20260929 16:01:15.668774 20241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36765 every 8 connection(s)
I20260929 16:01:15.670382 20094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:15.675827 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20094
I20260929 16:01:15.676609 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:44131
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=42765
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:01:15 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:15.702021 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.674018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:59837)
0929 16:01:15.674970 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.675020 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.677254 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.679102 (+ 1848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.679110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.679850 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:15.680538 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.680556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.683247 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.683254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.684936 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.684961 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.686797 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.686804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.687057 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.688147 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:15.688176 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:15.691280 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:15.696926 (+ 5646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:15.696935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:15.696953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:15.697623 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:15.698206 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:15.698209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:15.698211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:15.698420 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:15.699580 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:15.699586 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:15.699924 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.700254 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.700750 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":419,"threads_started":1}
I20260929 16:01:15.702332 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.673783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:59837 (local address 127.5.117.62:46489)
0929 16:01:15.674582 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.674594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.675168 (+ 574us) server_negotiation.cc:407] Connection header received
0929 16:01:15.677956 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.677976 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.678495 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.678960 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:15.680785 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.682859 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.685171 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.686440 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.686699 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.691695 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:15.691711 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:15.691742 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:15.691810 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:15.696698 (+ 4888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.697821 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.697835 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.697845 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.698085 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.698557 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.698565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.698571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.699428 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:15.699664 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.700533 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.701043 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":448,"threads_started":1}
I20260929 16:01:15.706385 20242 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:15.706801 20242 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:15.707902 20242 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:15.710595 20057 ts_manager.cc:194] Registered new tserver with Master: 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765)
I20260929 16:01:15.712317 20057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:59837
W20260929 16:01:16.049503 20246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:16.049779 20246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:16.049930 20246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:16.060998 20246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:16.061152 20246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:16.061343 20246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:01:16.072206 20246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:44131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=42765
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:16.073774 20246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:16.075292 20246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:16.090257 20251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:16.090330 20254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:16.090257 20252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:16.091395 20246 server_base.cc:1034] running on GCE node
I20260929 16:01:16.092038 20246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:16.093031 20246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:16.094265 20246 hybrid_clock.cc:648] HybridClock initialized: now 1790697676094230 us; error 50 us; skew 500 ppm
I20260929 16:01:16.094586 20246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:16 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:16.098304 20246 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:01:16.100385 20246 webserver.cc:460] Webserver started at http://127.5.117.2:42765/ using document root <none> and password file <none>
I20260929 16:01:16.101068 20246 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:16.101204 20246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:16.107923 20246 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 16:01:16.111056 20261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.112756 20246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:16.112931 20246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "f5c9344cbdbdac39f1695f971d365707"
server_key_iv: "b1a32c95d4dd489ac29ff4a9f5eee70e"
server_key_version: "encryptionkey@0"
I20260929 16:01:16.113965 20246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:16.138456 20246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:16.142675 20246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:16.142951 20246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:16.144037 20246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:16.145572 20246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:16.145659 20246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.145813 20246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:16.145905 20246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.179804 20246 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44131
I20260929 16:01:16.179906 20373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44131 every 8 connection(s)
I20260929 16:01:16.181334 20246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:16.182982 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20246
I20260929 16:01:16.183712 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:34939
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=33773
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:01:16 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:16.203846 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.184136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:46155 (local address 127.5.117.62:46489)
0929 16:01:16.184354 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.184361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.185613 (+ 1252us) server_negotiation.cc:407] Connection header received
0929 16:01:16.187611 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.187619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.187746 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.187877 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:16.189467 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.190609 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.191998 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.192807 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.192954 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.196493 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:16.196503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:16.196512 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:16.196561 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:16.200353 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.201008 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.201016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.201021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.201193 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.201648 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.201656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.201662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.202098 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:16.202284 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.203034 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.203343 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:01:16.204754 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.184598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:46155)
0929 16:01:16.185408 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.185459 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.187316 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.188110 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.188119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.188630 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:16.189292 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.189309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.190756 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.190760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.191855 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.191866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.193008 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.193013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.193200 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.193995 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:16.194017 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:16.196267 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:16.200473 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:16.200479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:16.200490 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:16.200880 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:16.201293 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:16.201296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:16.201298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:16.201510 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:16.202243 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:16.202249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:16.202581 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.202828 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.203386 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":342,"threads_started":1}
I20260929 16:01:16.206935 20374 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:16.207324 20374 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:16.208256 20374 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:16.210043 20057 ts_manager.cc:194] Registered new tserver with Master: d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131)
I20260929 16:01:16.210989 20057 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:46155
W20260929 16:01:16.558694 20377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:16.559019 20377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:16.559186 20377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:16.566826 20377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:16.566956 20377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:16.567170 20377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:01:16.577601 20377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=33773
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:16.579272 20377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:16.580906 20377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:16.595312 20382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:16.595371 20383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:16.595283 20385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:16.596808 20377 server_base.cc:1034] running on GCE node
I20260929 16:01:16.597433 20377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:16.598510 20377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:16.599764 20377 hybrid_clock.cc:648] HybridClock initialized: now 1790697676599724 us; error 57 us; skew 500 ppm
I20260929 16:01:16.600255 20377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:16 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:16.603603 20377 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:01:16.605261 20377 webserver.cc:460] Webserver started at http://127.5.117.3:33773/ using document root <none> and password file <none>
I20260929 16:01:16.606074 20377 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:16.606245 20377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:16.612227 20377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:16.614989 20392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.616325 20377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:16.616523 20377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "baea97ad1ca643c49d92e959cbfdc568"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-b69v"
server_key: "cfe4c4a578d4281cee9ec239459e741d"
server_key_iv: "f07a1409c5b028169672a92f0517df09"
server_key_version: "encryptionkey@0"
I20260929 16:01:16.617691 20377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:16.636801 20377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:16.640936 20377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:16.641219 20377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:16.642194 20377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:16.643555 20377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:16.643683 20377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.643829 20377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:16.643924 20377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:16.685860 20377 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34939
I20260929 16:01:16.685884 20504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34939 every 8 connection(s)
I20260929 16:01:16.687470 20377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:16.697856 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20377
Sep 29 16:01:16 dist-test-slave-b69v krb5kdc[19470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:16.713083 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.690717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:58343 (local address 127.5.117.62:46489)
0929 16:01:16.690963 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.690971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.692322 (+ 1351us) server_negotiation.cc:407] Connection header received
0929 16:01:16.694521 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.694526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.694613 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.694712 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:16.696339 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.697164 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.699445 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.700377 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.700547 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.705171 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:16.705185 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:16.705189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:16.705248 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:16.709044 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.709813 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.709819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.709823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.710016 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.710951 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.710958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.710963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.711427 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:16.711626 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.712661 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.712827 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:01:16.714636 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.691282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:58343)
0929 16:01:16.692160 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.692199 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.694168 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.694973 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.694982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.695507 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:16.696168 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.696187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.697387 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.697392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.699182 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.699204 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.700700 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.700711 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.701030 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.701836 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:16.701880 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:16.704904 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:16.709233 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:16.709241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:16.709251 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:16.709669 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:16.710249 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:16.710255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:16.710259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:16.710607 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:16.711614 (+ 1007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:16.711622 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:16.712173 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.712560 (+ 387us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.713239 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":347,"threads_started":1}
I20260929 16:01:16.715945 20242 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
I20260929 16:01:16.716331 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.698961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.5.117.62:46489)
0929 16:01:16.699520 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.699526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.700116 (+ 590us) server_negotiation.cc:407] Connection header received
0929 16:01:16.700330 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.700336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.700414 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.700529 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:16.702781 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.703658 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.705298 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.706027 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.706180 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.709230 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:16.709237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:16.709242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:16.709280 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:16.712362 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.713472 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.713476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.713478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.713590 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.714129 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.714136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.714139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.714544 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:16.714874 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.715533 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.715860 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":278,"threads_started":1}
I20260929 16:01:16.716583 20505 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46489
I20260929 16:01:16.717139 20505 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:16.718411 20505 heartbeater.cc:511] Master 127.5.117.62:46489 requested a full tablet report, sending...
I20260929 16:01:16.720361 20056 ts_manager.cc:194] Registered new tserver with Master: baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:16.721271 20056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:58343
I20260929 16:01:16.732003 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:16.738839 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.733059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.5.117.62:46489)
0929 16:01:16.733297 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.733306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.733679 (+ 373us) server_negotiation.cc:407] Connection header received
0929 16:01:16.733849 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.733853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.733987 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.734109 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:16.734516 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.735215 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.736785 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.737248 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.737342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.737773 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:16.738273 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:16.738388 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.738471 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.738533 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:01:16.739741 20056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:16.748026 20056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:16.772390 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.760766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34939 (local address 127.0.0.1:50484)
0929 16:01:16.762575 (+ 1809us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.762593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.762709 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.764431 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.764441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.764488 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:16.764948 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.764959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.766952 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.766957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.769070 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.769082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.771243 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.771250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.771448 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.771504 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.771633 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.771822 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1698,"mutex_wait_us":70,"thread_start_us":3038,"threads_started":1}
I20260929 16:01:16.772992 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.760441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:44131 (local address 127.0.0.1:33976)
0929 16:01:16.761588 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.761633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.761801 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.765413 (+ 3612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.765420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.765453 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:16.765863 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.765876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.768140 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.768148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.770203 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.770217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.772126 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.772133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.772320 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.772355 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.772472 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.772615 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":106,"thread_start_us":765,"threads_started":1}
I20260929 16:01:16.773052 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.760676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.5.117.2:44131)
0929 16:01:16.764799 (+ 4123us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.764807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.764833 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:01:16.764946 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.764954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.765206 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.765407 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:16.766057 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.767931 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.770367 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.771820 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.771993 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.772204 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.772541 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.772747 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3960,"thread_start_us":3838,"threads_started":1}
I20260929 16:01:16.773347 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.762538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.5.117.1:36765)
0929 16:01:16.763369 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.763381 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.764079 (+ 698us) server_negotiation.cc:407] Connection header received
0929 16:01:16.764194 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.764202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.764421 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.764626 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:16.765475 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.767049 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.769917 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.771461 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.771617 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.771872 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.772620 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.772797 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":429,"threads_started":1}
I20260929 16:01:16.773819 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.762033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39368)
0929 16:01:16.763776 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.763794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.763918 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.764777 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.764782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.764808 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:16.765214 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.765227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.767241 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.767247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.769755 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.769769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.771688 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.771695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.771857 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.771892 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.772020 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.772184 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1625,"mutex_wait_us":4,"thread_start_us":736,"threads_started":1}
I20260929 16:01:16.774151 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.761536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.5.117.3:34939)
0929 16:01:16.763726 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.763734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.763775 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:01:16.763900 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.763907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.764191 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.765124 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:16.765362 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.766795 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.769237 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.771013 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.773357 (+ 2344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.773564 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.773631 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.773777 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1979,"thread_start_us":2052,"threads_started":1}
I20260929 16:01:16.777165 20309 tablet_service.cc:1467] Processing CreateTablet for tablet d82e5e760d6c4e639f686602570a2b68 (DEFAULT_TABLE table=test-table [id=31a0ab4f129a49a5869eac05c49dea49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:16.777487 20440 tablet_service.cc:1467] Processing CreateTablet for tablet d82e5e760d6c4e639f686602570a2b68 (DEFAULT_TABLE table=test-table [id=31a0ab4f129a49a5869eac05c49dea49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:16.777477 20177 tablet_service.cc:1467] Processing CreateTablet for tablet d82e5e760d6c4e639f686602570a2b68 (DEFAULT_TABLE table=test-table [id=31a0ab4f129a49a5869eac05c49dea49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:16.778607 20309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82e5e760d6c4e639f686602570a2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:16.778852 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82e5e760d6c4e639f686602570a2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:16.778867 20177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d82e5e760d6c4e639f686602570a2b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:16.792228 20521 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Bootstrap starting.
I20260929 16:01:16.794771 20522 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Bootstrap starting.
I20260929 16:01:16.795620 20523 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Bootstrap starting.
I20260929 16:01:16.797485 20521 tablet_bootstrap.cc:654] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:16.798179 20522 tablet_bootstrap.cc:654] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:16.798735 20523 tablet_bootstrap.cc:654] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:16.799111 20521 log.cc:824] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:16.799388 20522 log.cc:824] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:16.799968 20523 log.cc:824] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:16.802588 20521 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: No bootstrap required, opened a new log
I20260929 16:01:16.802851 20521 ts_tablet_manager.cc:1397] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 16:01:16.803031 20523 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: No bootstrap required, opened a new log
I20260929 16:01:16.803274 20523 ts_tablet_manager.cc:1397] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 16:01:16.810335 20521 raft_consensus.cc:348] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.810696 20521 raft_consensus.cc:374] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:16.810444 20523 raft_consensus.cc:348] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.810798 20521 raft_consensus.cc:729] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d00aa67b3f4fa9a5c5ed30422780ec, State: Initialized, Role: FOLLOWER
I20260929 16:01:16.810839 20523 raft_consensus.cc:374] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:16.811002 20523 raft_consensus.cc:729] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80e1d8d4073a418baaa23fac399b6812, State: Initialized, Role: FOLLOWER
I20260929 16:01:16.811467 20521 consensus_queue.cc:260] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.811591 20523 consensus_queue.cc:260] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.812813 20374 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
I20260929 16:01:16.813360 20522 tablet_bootstrap.cc:492] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: No bootstrap required, opened a new log
I20260929 16:01:16.813632 20522 ts_tablet_manager.cc:1397] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20260929 16:01:16.814922 20521 ts_tablet_manager.cc:1428] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260929 16:01:16.816488 20523 ts_tablet_manager.cc:1428] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260929 16:01:16.823195 20522 raft_consensus.cc:348] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.823622 20522 raft_consensus.cc:374] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:16.823738 20522 raft_consensus.cc:729] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baea97ad1ca643c49d92e959cbfdc568, State: Initialized, Role: FOLLOWER
I20260929 16:01:16.824527 20522 consensus_queue.cc:260] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.825958 20505 heartbeater.cc:503] Master 127.5.117.62:46489 was elected leader, sending a full tablet report...
I20260929 16:01:16.826756 20522 ts_tablet_manager.cc:1428] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260929 16:01:16.923362 20243 tablet.cc:2369] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:16.933332 20375 tablet.cc:2369] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:16.939524 20506 tablet.cc:2369] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:16.942325 20528 raft_consensus.cc:482] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:16.942682 20528 raft_consensus.cc:504] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.947249 20528 leader_election.cc:288] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131), baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:16.953505 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.946135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:59359 (local address 127.5.117.2:44131)
0929 16:01:16.946427 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.946437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.947410 (+ 973us) server_negotiation.cc:407] Connection header received
0929 16:01:16.947722 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.947728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.947821 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.948039 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:16.948681 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.949477 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.951425 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.952400 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.952574 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.952690 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.953055 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.953190 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:01:16.953505 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.946392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:50119 (local address 127.5.117.3:34939)
0929 16:01:16.946671 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.946680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.947941 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 16:01:16.948252 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.948262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.948328 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.948427 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:16.949004 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.949754 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.951534 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.952458 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.952616 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.952739 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.953077 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.953210 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:01:16.953647 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.946115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:34939 (local address 127.5.117.1:50119)
0929 16:01:16.947835 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.947849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.948001 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.948457 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.948462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.948492 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:16.948852 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.948861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.949898 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.949903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.951366 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.951378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.952692 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.952698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.952911 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.952934 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.953049 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.953169 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1631,"mutex_wait_us":29,"thread_start_us":788,"threads_started":1}
I20260929 16:01:16.954073 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.946033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:44131 (local address 127.5.117.1:59359)
0929 16:01:16.947277 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:16.947307 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:16.947516 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:16.948076 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:16.948082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:16.948112 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:16.948525 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.948535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.949621 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.949629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.951260 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:16.951273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.952648 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.952656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:16.952871 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.952897 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:01:16.953024 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:01:16.953160 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":21,"thread_start_us":1556,"threads_started":1}
I20260929 16:01:16.954779 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82e5e760d6c4e639f686602570a2b68" candidate_uuid: "80e1d8d4073a418baaa23fac399b6812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baea97ad1ca643c49d92e959cbfdc568" is_pre_election: true
I20260929 16:01:16.954942 20329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82e5e760d6c4e639f686602570a2b68" candidate_uuid: "80e1d8d4073a418baaa23fac399b6812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" is_pre_election: true
I20260929 16:01:16.955376 20460 raft_consensus.cc:2454] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e1d8d4073a418baaa23fac399b6812 in term 0.
I20260929 16:01:16.955466 20329 raft_consensus.cc:2454] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80e1d8d4073a418baaa23fac399b6812 in term 0.
I20260929 16:01:16.956321 20131 leader_election.cc:302] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80e1d8d4073a418baaa23fac399b6812, baea97ad1ca643c49d92e959cbfdc568, d6d00aa67b3f4fa9a5c5ed30422780ec; no voters:
I20260929 16:01:16.956748 20528 raft_consensus.cc:2781] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:16.956910 20528 raft_consensus.cc:482] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:16.957058 20528 raft_consensus.cc:3037] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:16.959584 20528 raft_consensus.cc:504] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.960322 20528 leader_election.cc:288] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [CANDIDATE]: Term 1 election: Requested vote from peers d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131), baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939)
I20260929 16:01:16.960665 20329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82e5e760d6c4e639f686602570a2b68" candidate_uuid: "80e1d8d4073a418baaa23fac399b6812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
I20260929 16:01:16.960877 20329 raft_consensus.cc:3037] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:16.961027 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d82e5e760d6c4e639f686602570a2b68" candidate_uuid: "80e1d8d4073a418baaa23fac399b6812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baea97ad1ca643c49d92e959cbfdc568"
I20260929 16:01:16.961256 20460 raft_consensus.cc:3037] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:16.964505 20329 raft_consensus.cc:2454] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e1d8d4073a418baaa23fac399b6812 in term 1.
I20260929 16:01:16.964715 20460 raft_consensus.cc:2454] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80e1d8d4073a418baaa23fac399b6812 in term 1.
I20260929 16:01:16.964964 20131 leader_election.cc:302] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80e1d8d4073a418baaa23fac399b6812, d6d00aa67b3f4fa9a5c5ed30422780ec; no voters:
I20260929 16:01:16.965301 20528 raft_consensus.cc:2781] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:16.965762 20528 raft_consensus.cc:686] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 1 LEADER]: Becoming Leader. State: Replica: 80e1d8d4073a418baaa23fac399b6812, State: Running, Role: LEADER
I20260929 16:01:16.966434 20528 consensus_queue.cc:237] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } }
I20260929 16:01:16.971647 20057 catalog_manager.cc:5495] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1). New cstate: current_term: 1 leader_uuid: "80e1d8d4073a418baaa23fac399b6812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80e1d8d4073a418baaa23fac399b6812" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36765 } health_report { overall_health: HEALTHY } } }
I20260929 16:01:17.021051 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.013716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.5.117.1:36765)
0929 16:01:17.013989 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:17.013994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:17.014015 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:01:17.014254 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:17.014264 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:17.014389 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:17.014575 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:17.015045 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.015835 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.018275 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.018825 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.019083 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:17.019652 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:17.020324 (+ 672us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:17.020479 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:17.020613 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 16:01:17.020696 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:01:17.030799 20460 raft_consensus.cc:1261] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Refusing update from remote peer 80e1d8d4073a418baaa23fac399b6812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:17.030793 20329 raft_consensus.cc:1261] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Refusing update from remote peer 80e1d8d4073a418baaa23fac399b6812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:17.031839 20528 consensus_queue.cc:1035] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baea97ad1ca643c49d92e959cbfdc568" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 34939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:17.032161 20532 consensus_queue.cc:1035] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:17.050510 20536 mvcc.cc:205] Tried to move back new op lower bound from 7334697685106016256 to 7334697684858744832. Current Snapshot: MvccSnapshot[applied={T|T < 7334697685106016256}]
I20260929 16:01:17.065312 20056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:17.065645 20056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:17.069134 20056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Sending DeleteTablet for 3 replicas of tablet d82e5e760d6c4e639f686602570a2b68
I20260929 16:01:17.070277 20309 tablet_service.cc:1514] Processing DeleteTablet for tablet d82e5e760d6c4e639f686602570a2b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:17 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:33976
I20260929 16:01:17.070380 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet d82e5e760d6c4e639f686602570a2b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:17 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:50484
I20260929 16:01:17.070654 20177 tablet_service.cc:1514] Processing DeleteTablet for tablet d82e5e760d6c4e639f686602570a2b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:17 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:39368
I20260929 16:01:17.070765 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20094
I20260929 16:01:17.072613 20545 tablet_replica.cc:330] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: stopping tablet replica
I20260929 16:01:17.073235 20547 tablet_replica.cc:330] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: stopping tablet replica
I20260929 16:01:17.073518 20546 tablet_replica.cc:330] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: stopping tablet replica
I20260929 16:01:17.073680 20545 raft_consensus.cc:2229] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:17.073784 20547 raft_consensus.cc:2229] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:17.074258 20546 raft_consensus.cc:2229] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:17.074471 20545 raft_consensus.cc:2258] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:17.074497 20547 raft_consensus.cc:2258] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:17.075069 20546 raft_consensus.cc:2258] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:17.077049 20547 ts_tablet_manager.cc:1905] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:17.077967 20546 ts_tablet_manager.cc:1905] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:17.081821 20545 ts_tablet_manager.cc:1905] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:17.086736 20546 ts_tablet_manager.cc:1918] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:17.087024 20546 log.cc:1196] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/d82e5e760d6c4e639f686602570a2b68
I20260929 16:01:17.087831 20546 ts_tablet_manager.cc:1939] T d82e5e760d6c4e639f686602570a2b68 P baea97ad1ca643c49d92e959cbfdc568: Deleting consensus metadata
I20260929 16:01:17.090617 20043 catalog_manager.cc:4841] TS baea97ad1ca643c49d92e959cbfdc568 (127.5.117.3:34939): tablet d82e5e760d6c4e639f686602570a2b68 (table test-table [id=31a0ab4f129a49a5869eac05c49dea49]) successfully deleted
I20260929 16:01:17.093976 20545 ts_tablet_manager.cc:1918] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:17.094195 20545 log.cc:1196] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/d82e5e760d6c4e639f686602570a2b68
I20260929 16:01:17.094642 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.071708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.5.117.1:36765)
0929 16:01:17.072326 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:17.072336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:17.072356 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:01:17.072436 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:17.072441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:17.072545 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:17.072652 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:17.074909 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.076387 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.078961 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.079784 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.080103 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:17.084124 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:17.084138 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:17.084156 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:17.084211 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:17.088787 (+ 4576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.090366 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.090390 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.090397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.090664 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.091281 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.091287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.091291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.091878 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:17.092267 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:17.093793 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0929 16:01:17.094113 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260929 16:01:17.094832 20545 ts_tablet_manager.cc:1939] T d82e5e760d6c4e639f686602570a2b68 P d6d00aa67b3f4fa9a5c5ed30422780ec: Deleting consensus metadata
I20260929 16:01:17.095007 20547 ts_tablet_manager.cc:1918] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:17.095261 20547 log.cc:1196] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/d82e5e760d6c4e639f686602570a2b68
I20260929 16:01:17.095534 20237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:17.097438 20042 catalog_manager.cc:4841] TS d6d00aa67b3f4fa9a5c5ed30422780ec (127.5.117.2:44131): tablet d82e5e760d6c4e639f686602570a2b68 (table test-table [id=31a0ab4f129a49a5869eac05c49dea49]) successfully deleted
I20260929 16:01:17.224128 20547 ts_tablet_manager.cc:1939] T d82e5e760d6c4e639f686602570a2b68 P 80e1d8d4073a418baaa23fac399b6812: Deleting consensus metadata
I20260929 16:01:17.224987 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20094
W20260929 16:01:17.247928 20042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.117.1:36765 (error 108)
I20260929 16:01:17.248178 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20246
I20260929 16:01:17.250013 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.249499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39398)
0929 16:01:17.249675 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.249801 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:01:17.250240 20042 catalog_manager.cc:4584] TS 80e1d8d4073a418baaa23fac399b6812 (127.5.117.1:36765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d82e5e760d6c4e639f686602570a2b68: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
I20260929 16:01:17.261574 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.249086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.5.117.2:44131)
0929 16:01:17.249273 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:17.249277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:17.249295 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:01:17.249506 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:17.249512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:17.249625 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:17.249728 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:17.251227 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.251960 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.253485 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.253871 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.253959 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:17.256663 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:17.256673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:17.256689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:17.256718 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:17.259351 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.259909 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.259915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.259917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.260029 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.260372 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.260375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.260376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.260751 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:17.260967 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:17.261172 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 16:01:17.261325 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:01:17.262183 20369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:17.304369 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.303672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39414)
0929 16:01:17.304029 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.304190 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260929 16:01:17.305625 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.305140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39420)
0929 16:01:17.305299 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.305425 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:01:17.357627 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20246
I20260929 16:01:17.366482 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.365761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39436)
0929 16:01:17.366001 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.366132 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:01:17.367879 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.367525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39446)
0929 16:01:17.367707 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.367777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:01:17.379951 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20377
I20260929 16:01:17.394892 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.380836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.5.117.3:34939)
0929 16:01:17.381068 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:17.381073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:17.381094 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:01:17.381200 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:17.381204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:17.381279 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:17.381359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:17.382764 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.383566 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.385252 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:17.385738 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:17.385917 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:17.388756 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:17.388771 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:17.388788 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:17.388850 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:17.392235 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.392963 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.392972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.392974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.393142 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:17.393472 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:17.393475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:17.393477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:17.393855 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:17.394118 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:17.394296 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 16:01:17.394484 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 16:01:17.395596 20500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:17.449388 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.448800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39456)
0929 16:01:17.449062 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.449169 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:01:17.450596 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.450206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39460)
0929 16:01:17.450417 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.450498 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 16:01:17.489269 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20377
I20260929 16:01:17.509493 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20024
I20260929 16:01:17.510788 20087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:17.634410 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:17.633651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36765 (local address 127.0.0.1:39476)
0929 16:01:17.633948 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:17.634091 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:36765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:01:17.634522 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20024
I20260929 16:01:17.662458 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--webserver_interface=127.5.117.62
--webserver_port=40345
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:18.051744 20553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:18.052112 20553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:18.052278 20553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:18.064030 20553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:18.064191 20553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:18.064260 20553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:18.064322 20553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:18.064368 20553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:18.073580 20553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=40345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:18.074852 20553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:18.076176 20553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:18.088115 20559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:18.088562 20561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:18.088900 20553 server_base.cc:1034] running on GCE node
W20260929 16:01:18.089087 20558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:18.089901 20553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:18.091246 20553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:18.092481 20553 hybrid_clock.cc:648] HybridClock initialized: now 1790697678092436 us; error 53 us; skew 500 ppm
I20260929 16:01:18.092845 20553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:18 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697678, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:18.095841 20553 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:18.097549 20553 webserver.cc:460] Webserver started at http://127.5.117.62:40345/ using document root <none> and password file <none>
I20260929 16:01:18.098320 20553 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:18.098488 20553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:18.104790 20553 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:18.109737 20568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:18.111294 20553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:01:18.111598 20553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "37e7517170d543de8e6ce939a0140999"
format_stamp: "Formatted at 2026-09-29 16:01:11 on dist-test-slave-b69v"
server_key: "3b426ece99f250a98c08372e67a23206"
server_key_iv: "53d9af8043a919a5633a23b6b7375017"
server_key_version: "encryptionkey@0"
I20260929 16:01:18.112869 20553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:18.142390 20553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:18.147401 20553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:18.147749 20553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:18.170621 20553 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46489
I20260929 16:01:18.170722 20619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46489 every 8 connection(s)
I20260929 16:01:18.172390 20553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:18.179942 20620 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:01:18.181389 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20553
I20260929 16:01:18.182507 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:36765
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=42877
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:18.184772 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap starting.
I20260929 16:01:18.211665 20620 log.cc:824] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:18.228581 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:01:18.229094 20620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999: Bootstrap complete.
I20260929 16:01:18.238184 20620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:18.239032 20620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Initialized, Role: FOLLOWER
I20260929 16:01:18.239648 20620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:18.239889 20620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:18.240032 20620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:18.240263 20620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:01:18.244065 20620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:18.244659 20620 leader_election.cc:302] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e7517170d543de8e6ce939a0140999; no voters:
I20260929 16:01:18.245438 20620 leader_election.cc:288] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:01:18.245563 20624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:01:18.246945 20624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [term 3 LEADER]: Becoming Leader. State: Replica: 37e7517170d543de8e6ce939a0140999, State: Running, Role: LEADER
I20260929 16:01:18.247602 20624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } }
I20260929 16:01:18.247985 20620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:18.250232 20626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e7517170d543de8e6ce939a0140999. Latest consensus state: current_term: 3 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:18.250653 20626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:18.251291 20625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37e7517170d543de8e6ce939a0140999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e7517170d543de8e6ce939a0140999" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46489 } } }
I20260929 16:01:18.251619 20625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e7517170d543de8e6ce939a0140999 [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:18.260079 20630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:18.270879 20630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:18.272624 20630 catalog_manager.cc:1259] Loaded cluster ID: 2c470e4e6c0c4a5f8213dfe808f813f6
I20260929 16:01:18.272778 20630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:01:18.275031 20630 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0bfd13d9ad google::LogMessage::Fail() at ??:0
@ 0x7f0bfd14088a google::LogMessage::SendToLog() at ??:0
@ 0x7f0bfd13d4de google::LogMessage::Flush() at ??:0
@ 0x7f0bfd13ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0c12b70512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0c12b6d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0c12bf5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0c12bf5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0bffd89345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0bffd8c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0bffd8c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0bffd67eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0c030416db start_thread at ??:0
@ 0x7f0bfbfd761f clone at ??:0
W20260929 16:01:18.646221 20622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:18.646541 20622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:18.646638 20622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:18.653322 20622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:18.653419 20622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:18.653543 20622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:01:18.662739 20622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:36765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=42877
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:18.664054 20622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:18.665421 20622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:18.678979 20658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:18.679070 20659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:18.679026 20661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:18.680816 20622 server_base.cc:1034] running on GCE node
I20260929 16:01:18.681476 20622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:18.682600 20622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:18.683817 20622 hybrid_clock.cc:648] HybridClock initialized: now 1790697678683770 us; error 42 us; skew 500 ppm
I20260929 16:01:18.684204 20622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:18 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697678, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:18.687966 20622 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:01:18.689891 20622 webserver.cc:460] Webserver started at http://127.5.117.1:42877/ using document root <none> and password file <none>
I20260929 16:01:18.690775 20622 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:18.690922 20622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:18.697639 20622 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:01:18.701110 20668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:18.702948 20622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:01:18.703164 20622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "80e1d8d4073a418baaa23fac399b6812"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "0787c0ca4239cffbd81f7bcb09c1ab5c"
server_key_iv: "ecbdd27ab0e475054c0dbbd30564a833"
server_key_version: "encryptionkey@0"
I20260929 16:01:18.704485 20622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:18.725697 20622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:18.730346 20622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:18.730672 20622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:18.731799 20622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:18.733352 20622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:18.733491 20622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:18.733644 20622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:18.733742 20622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:18.774937 20622 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36765
I20260929 16:01:18.774992 20780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36765 every 8 connection(s)
I20260929 16:01:18.776824 20622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:18.782464 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:18.780418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:33385)
0929 16:01:18.781229 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:18.781462 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
W20260929 16:01:18.783161 20781 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:46489 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:46489: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
I20260929 16:01:18.783989 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:18.783685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:43093)
0929 16:01:18.783818 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:18.783886 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:01:18.784806 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:18.784515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:58277)
0929 16:01:18.784633 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:18.784702 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:01:18.785053 20781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:01:18.787256 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20622
I20260929 16:01:18.787983 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:44131
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=42765
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:19.180186 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:19.180589 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:19.180778 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:19.188194 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:19.188339 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:19.188560 20784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:01:19.199402 20784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:44131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=42765
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:19.201277 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:19.202916 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:19.218708 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:19.218627 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:19.218966 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:19.219547 20784 server_base.cc:1034] running on GCE node
I20260929 16:01:19.220321 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:19.221582 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:19.222810 20784 hybrid_clock.cc:648] HybridClock initialized: now 1790697679222758 us; error 58 us; skew 500 ppm
I20260929 16:01:19.223163 20784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:19 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:19.227065 20784 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:01:19.229031 20784 webserver.cc:460] Webserver started at http://127.5.117.2:42765/ using document root <none> and password file <none>
I20260929 16:01:19.229840 20784 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:19.229956 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:19.235924 20784 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 16:01:19.238900 20799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.240461 20784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:19.240648 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "d6d00aa67b3f4fa9a5c5ed30422780ec"
format_stamp: "Formatted at 2026-09-29 16:01:12 on dist-test-slave-b69v"
server_key: "f5c9344cbdbdac39f1695f971d365707"
server_key_iv: "b1a32c95d4dd489ac29ff4a9f5eee70e"
server_key_version: "encryptionkey@0"
I20260929 16:01:19.241735 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:19.255533 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:19.259599 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:19.259886 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:19.260931 20784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:19.262326 20784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:19.262454 20784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.262598 20784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:19.262693 20784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.300585 20784 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44131
I20260929 16:01:19.300650 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44131 every 8 connection(s)
I20260929 16:01:19.302255 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:19.307739 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.305733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:49161)
0929 16:01:19.306594 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.306813 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":396,"threads_started":1}
W20260929 16:01:19.308550 20912 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:46489 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:46489: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
I20260929 16:01:19.309356 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.309014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:54341)
0929 16:01:19.309167 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.309246 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:01:19.310207 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.309858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:35867)
0929 16:01:19.309998 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.310094 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:01:19.310645 20912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:01:19.310971 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20784
I20260929 16:01:19.311667 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:34939
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:46489
--webserver_port=33773
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:19.669562 20915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:19.669842 20915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:19.669937 20915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:19.676847 20915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:19.676940 20915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:19.677053 20915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:01:19.686195 20915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:34939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=33773
--tserver_master_addrs=127.5.117.62:46489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:19.687382 20915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:19.688740 20915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:19.702701 20920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:19.702754 20923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:19.702786 20921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:19.703896 20915 server_base.cc:1034] running on GCE node
I20260929 16:01:19.704520 20915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:19.705680 20915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:19.706950 20915 hybrid_clock.cc:648] HybridClock initialized: now 1790697679706901 us; error 44 us; skew 500 ppm
I20260929 16:01:19.707295 20915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:19 dist-test-slave-b69v krb5kdc[19470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697679, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:19.710966 20915 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:01:19.712635 20915 webserver.cc:460] Webserver started at http://127.5.117.3:33773/ using document root <none> and password file <none>
I20260929 16:01:19.713366 20915 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:19.713518 20915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:19.719273 20915 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:01:19.722069 20930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.723371 20915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:01:19.723543 20915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "baea97ad1ca643c49d92e959cbfdc568"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-b69v"
server_key: "cfe4c4a578d4281cee9ec239459e741d"
server_key_iv: "f07a1409c5b028169672a92f0517df09"
server_key_version: "encryptionkey@0"
I20260929 16:01:19.724607 20915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:19.762717 20915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:19.766880 20915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:19.767171 20915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:19.768326 20915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:19.769905 20915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:19.770052 20915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.770202 20915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:19.770298 20915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:19.787526 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.786130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:38455)
0929 16:01:19.786915 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.787083 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":390,"threads_started":1}
I20260929 16:01:19.810310 20915 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:34939
I20260929 16:01:19.810400 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:34939 every 8 connection(s)
I20260929 16:01:19.811924 20915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:19.817309 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.815235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:35813)
0929 16:01:19.816109 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.816365 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
W20260929 16:01:19.818486 21044 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:46489 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:46489: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
I20260929 16:01:19.819646 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.819284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:53635)
0929 16:01:19.819449 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.819529 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:01:19.820591 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 20915
I20260929 16:01:19.820791 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:19.820427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:55433)
0929 16:01:19.820561 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:19.820638 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:01:19.821089 21044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:01:20.312316 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:20.311238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:47807)
0929 16:01:20.311820 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:20.311958 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
I20260929 16:01:20.789968 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:20.788946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:39583)
0929 16:01:20.789624 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:20.789749 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":352,"threads_started":1}
I20260929 16:01:20.822769 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:20.821751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:49937)
0929 16:01:20.822362 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:20.822495 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":344,"threads_started":1}
I20260929 16:01:21.314311 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:21.313341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:47605)
0929 16:01:21.313962 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:21.314050 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":351,"threads_started":1}
I20260929 16:01:21.791973 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:21.791113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:43137)
0929 16:01:21.791642 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:21.791741 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":280,"threads_started":1}
I20260929 16:01:21.824779 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:21.823858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:34097)
0929 16:01:21.824420 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:21.824514 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":330,"threads_started":1}
I20260929 16:01:22.316267 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:22.315291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:40603)
0929 16:01:22.315918 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:22.316009 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":365,"threads_started":1}
I20260929 16:01:22.794389 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:22.793274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:37795)
0929 16:01:22.794037 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:22.794149 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":337,"threads_started":1}
I20260929 16:01:22.826539 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:22.825805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:37215)
0929 16:01:22.826277 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:22.826360 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260929 16:01:23.318107 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:23.317292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:42533)
0929 16:01:23.317773 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:23.317872 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260929 16:01:23.796666 21058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:23.795707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:42913)
0929 16:01:23.796359 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:23.796458 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":415,"threads_started":1}
I20260929 16:01:23.828514 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:23.827560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:36247)
0929 16:01:23.828180 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:23.828289 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
I20260929 16:01:24.320513 21060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:24.319448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:46473)
0929 16:01:24.320133 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:24.320284 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":340,"threads_started":1}
I20260929 16:01:24.798465 21061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:24.797614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:36605)
0929 16:01:24.798136 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:24.798249 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
I20260929 16:01:24.830283 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:24.829440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:47801)
0929 16:01:24.830003 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:24.830087 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":311,"threads_started":1}
I20260929 16:01:25.322958 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:25.321841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:45523)
0929 16:01:25.322587 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:25.322730 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":453,"threads_started":1}
I20260929 16:01:25.800550 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:25.799564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:40477)
0929 16:01:25.800204 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:25.800293 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":259,"threads_started":1}
I20260929 16:01:25.832182 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:25.831255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:39239)
0929 16:01:25.831798 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:25.831904 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
I20260929 16:01:26.325207 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:26.324232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:58281)
0929 16:01:26.324842 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:26.324949 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":244,"threads_started":1}
I20260929 16:01:26.802485 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:26.801683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:45739)
0929 16:01:26.802180 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:26.802268 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260929 16:01:26.834540 21068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:26.833494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:60185)
0929 16:01:26.834147 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:26.834269 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":274,"threads_started":1}
I20260929 16:01:27.327375 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:27.326398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:43031)
0929 16:01:27.327033 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:27.327134 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":359,"threads_started":1}
I20260929 16:01:27.804735 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:27.803713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:55801)
0929 16:01:27.804404 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:27.804509 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":387,"threads_started":1}
I20260929 16:01:27.837241 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:27.836192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:53395)
0929 16:01:27.836908 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:27.837012 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":449,"threads_started":1}
I20260929 16:01:28.329984 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:28.328849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:52661)
0929 16:01:28.329651 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:28.329758 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":478,"threads_started":1}
I20260929 16:01:28.806969 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:28.805957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:40585)
0929 16:01:28.806569 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:28.806678 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":332,"threads_started":1}
I20260929 16:01:28.839052 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:28.838213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:49127)
0929 16:01:28.838731 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:28.838831 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":261,"threads_started":1}
I20260929 16:01:29.332455 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:29.331299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:44397)
0929 16:01:29.332007 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:29.332177 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":419,"threads_started":1}
I20260929 16:01:29.809131 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:29.808185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:58665)
0929 16:01:29.808714 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:29.808886 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
I20260929 16:01:29.841195 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:29.840324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:44865)
0929 16:01:29.840878 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:29.840986 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":288,"threads_started":1}
I20260929 16:01:30.334342 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:30.333480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:42541)
0929 16:01:30.333978 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:30.334068 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":240,"threads_started":1}
I20260929 16:01:30.811331 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:30.810315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:52651)
0929 16:01:30.810978 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:30.811105 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":354,"threads_started":1}
I20260929 16:01:30.843576 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:30.842444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:33743)
0929 16:01:30.843211 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:30.843356 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":302,"threads_started":1}
I20260929 16:01:31.336581 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:31.335549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:49525)
0929 16:01:31.336242 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:31.336382 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":341,"threads_started":1}
I20260929 16:01:31.813340 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:31.812463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:55007)
0929 16:01:31.813038 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:31.813135 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":324,"threads_started":1}
I20260929 16:01:31.845830 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:31.845002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:35315)
0929 16:01:31.845531 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:31.845609 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":303,"threads_started":1}
I20260929 16:01:32.338694 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:32.337661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:57337)
0929 16:01:32.338351 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:32.338456 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":423,"threads_started":1}
I20260929 16:01:32.815104 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:32.814294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:36079)
0929 16:01:32.814789 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:32.814910 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":247,"threads_started":1}
I20260929 16:01:32.848196 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:32.847052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:46069)
0929 16:01:32.847716 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:32.847861 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":309,"threads_started":1}
I20260929 16:01:33.340529 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:33.339671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:54385)
0929 16:01:33.340214 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:33.340300 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
I20260929 16:01:33.817308 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:33.816413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:57011)
0929 16:01:33.817013 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:33.817115 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":353,"threads_started":1}
I20260929 16:01:33.850069 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:33.849335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:57585)
0929 16:01:33.849784 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:33.849878 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260929 16:01:34.342877 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:34.341812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.2:34815)
0929 16:01:34.342450 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:34.342637 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":325,"threads_started":1}
I20260929 16:01:34.818972 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:34.818211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.1:51649)
0929 16:01:34.818723 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:34.818812 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260929 16:01:34.827531 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20622
I20260929 16:01:34.852397 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:34.851374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46489 (local address 127.5.117.3:50899)
0929 16:01:34.851982 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:34.852165 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:46489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":342,"threads_started":1}
I20260929 16:01:34.854925 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:34.828748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.5.117.1:36765)
0929 16:01:34.829435 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:34.829453 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:34.829527 (+ 74us) server_negotiation.cc:407] Connection header received
0929 16:01:34.831555 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:34.831566 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:34.832143 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:34.832516 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:34.834000 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:34.835211 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:34.836444 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:34.837067 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:34.837199 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:34.847252 (+ 10053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:34.847263 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:34.847286 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:34.847334 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:34.850808 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:34.851756 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:34.851772 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:34.851781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:34.851974 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:34.852449 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:34.852455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:34.852458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:34.853247 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:34.853605 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:34.854084 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 16:01:34.854620 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":320,"threads_started":1}
I20260929 16:01:34.856514 20776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:34.949254 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20622
I20260929 16:01:34.972514 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20784
I20260929 16:01:34.992834 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:34.973715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.5.117.2:44131)
0929 16:01:34.974339 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:34.974358 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:34.974407 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:01:34.976281 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:34.976292 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:34.976824 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:34.977162 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:34.978542 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:34.979919 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:34.981544 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:34.982577 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:34.982836 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:34.985680 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:34.985690 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:34.985708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:34.985758 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:34.989412 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:34.990161 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:34.990168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:34.990174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:34.990383 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:34.990733 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:34.990736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:34.990738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:34.991319 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:34.991584 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:34.992016 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 16:01:34.992491 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260929 16:01:34.994347 20907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:35.085233 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20784
I20260929 16:01:35.106666 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 20915
I20260929 16:01:35.124991 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:35.107781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.5.117.3:34939)
0929 16:01:35.108504 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:35.108522 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:35.108573 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:01:35.110452 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:35.110462 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:35.110965 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:35.111303 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:35.112928 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:35.114201 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:35.115465 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:35.116183 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:35.116312 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:35.119109 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:35.119116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:35.119131 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:35.119166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:35.122023 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:35.122628 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:35.122634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:35.122640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:35.122808 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:35.123134 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:35.123137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:35.123138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:35.123654 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:35.123868 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:35.124272 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 16:01:35.124688 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":360,"threads_started":1}
I20260929 16:01:35.126416 21039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:35.219087 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 20915
2026-09-29T16:01:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25508 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:01:35 dist-test-slave-b69v krb5kdc[21101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:01:35 dist-test-slave-b69v krb5kdc[21101](info): set up 2 sockets
Sep 29 16:01:35 dist-test-slave-b69v krb5kdc[21101](info): commencing operation
krb5kdc: starting...
W20260929 16:01:36.345063 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:01:36 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:01:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:01:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:01:36.633128 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:36.996658 21114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:36.997107 21114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:36.997244 21114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:37.004997 21114 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:01:37.005092 21114 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:01:37.005134 21114 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:01:37.005226 21114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:37.005334 21114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:37.005467 21114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:37.005553 21114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:37.005643 21114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:37.015674 21114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:37.017140 21114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:37.018625 21114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:37.032063 21119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.031998 21122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.032141 21120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:37.033020 21114 server_base.cc:1034] running on GCE node
I20260929 16:01:37.033869 21114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:37.035492 21114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:37.036756 21114 hybrid_clock.cc:648] HybridClock initialized: now 1790697697036699 us; error 34 us; skew 500 ppm
I20260929 16:01:37.037221 21114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:37 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:37.041600 21114 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:01:37.043154 21114 webserver.cc:460] Webserver started at http://127.5.117.62:46335/ using document root <none> and password file <none>
I20260929 16:01:37.043879 21114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:37.044021 21114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:37.044394 21114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:37.047528 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "fd82de88e6dd4e6393aa16e939b49bcb"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "8f4901ae47d34c33484de88f6fc55760"
server_key_iv: "d5aca71fbc24bd36935848e00ed19138"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.048506 21114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "fd82de88e6dd4e6393aa16e939b49bcb"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "8f4901ae47d34c33484de88f6fc55760"
server_key_iv: "d5aca71fbc24bd36935848e00ed19138"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.055231 21114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 16:01:37.059264 21129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.060827 21114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:01:37.061045 21114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "fd82de88e6dd4e6393aa16e939b49bcb"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "8f4901ae47d34c33484de88f6fc55760"
server_key_iv: "d5aca71fbc24bd36935848e00ed19138"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.061357 21114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:37.093524 21114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:37.098352 21114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:37.098664 21114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:37.119176 21114 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39343
I20260929 16:01:37.119215 21180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39343 every 8 connection(s)
I20260929 16:01:37.121166 21114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:37.125335 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:37.131042 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21114
I20260929 16:01:37.131337 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:01:37.132148 5588 external_mini_cluster.cc:1428] Setting key a5632b846df966196267c2a545ef7d4a
I20260929 16:01:37.137224 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap starting.
I20260929 16:01:37.141283 21181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:37.143277 21181 log.cc:824] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:37.146809 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: No bootstrap required, opened a new log
Sep 29 16:01:37 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:37.155680 21181 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:01:37.156106 21181 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:37.156248 21181 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Initialized, Role: FOLLOWER
I20260929 16:01:37.156924 21181 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:01:37.157225 21181 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:37.157389 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:37.157588 21181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:37.157871 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.134302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.5.117.62:39343)
0929 16:01:37.135302 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:37.135314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:37.135359 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:01:37.137264 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:37.137287 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:37.137703 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:37.138031 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:37.140154 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.141532 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.144128 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.145079 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.145210 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.150297 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:37.150305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:37.150318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:37.150358 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:37.153551 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.154289 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.154296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.154301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.154485 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.154911 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.154917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.154920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.155450 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:37.155762 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:37.156216 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:01:37.156667 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":474,"threads_started":1}
I20260929 16:01:37.160254 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:01:37.160866 21181 leader_election.cc:302] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd82de88e6dd4e6393aa16e939b49bcb; no voters:
I20260929 16:01:37.161406 21181 leader_election.cc:288] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:01:37.161566 21186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:37.162736 21186 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 LEADER]: Becoming Leader. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Running, Role: LEADER
I20260929 16:01:37.163584 21186 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:01:37.163921 21181 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:37.170141 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd82de88e6dd4e6393aa16e939b49bcb. Latest consensus state: current_term: 1 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:01:37.170718 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:37.171276 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:01:37.171636 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:37.171859 21196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:37.178416 21196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:37.190644 21196 catalog_manager.cc:1347] Generated new cluster ID: 56f52f1a96c54476bb7c2957200260f5
I20260929 16:01:37.190778 21196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:37.203469 21196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:01:37.205120 21196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:37.221827 21196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Generated new TSK 0
I20260929 16:01:37.222693 21196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:01:37.299135 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:37.686483 21207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:37.686972 21207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:37.687299 21207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:37.696295 21207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:37.696456 21207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:37.696651 21207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:01:37.706708 21207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:37.708151 21207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:37.709626 21207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:37.724095 21213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.724223 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.724079 21215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:37.725061 21207 server_base.cc:1034] running on GCE node
I20260929 16:01:37.725746 21207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:37.726701 21207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:37.727988 21207 hybrid_clock.cc:648] HybridClock initialized: now 1790697697727927 us; error 56 us; skew 500 ppm
I20260929 16:01:37.728343 21207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:37 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:37.731716 21207 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:01:37.733337 21207 webserver.cc:460] Webserver started at http://127.5.117.1:36427/ using document root <none> and password file <none>
I20260929 16:01:37.734078 21207 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:37.734248 21207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:37.734571 21207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:37.737643 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "940d9d1e4884205c6ff808c3cfdfc1c6"
server_key_iv: "1524151bc912abcf781b48acefd1f5bf"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.738497 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "940d9d1e4884205c6ff808c3cfdfc1c6"
server_key_iv: "1524151bc912abcf781b48acefd1f5bf"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.743712 21207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:01:37.747228 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.748623 21207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:01:37.748806 21207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "940d9d1e4884205c6ff808c3cfdfc1c6"
server_key_iv: "1524151bc912abcf781b48acefd1f5bf"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.749006 21207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:37.767899 21207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:37.772904 21207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:37.773247 21207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:37.774417 21207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:37.775983 21207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:37.776124 21207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:01:37.776291 21207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:37.776369 21207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.821132 21334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36091 every 8 connection(s)
I20260929 16:01:37.821132 21207 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36091
I20260929 16:01:37.823995 21207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:37.828032 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21207
I20260929 16:01:37.828320 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:01:37.829274 5588 external_mini_cluster.cc:1428] Setting key be27b73462ae0a7645d222e9e5f5ebec
Sep 29 16:01:37 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:37.854291 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.827683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:48867 (local address 127.5.117.62:39343)
0929 16:01:37.828247 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:37.828254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:37.829577 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 16:01:37.832106 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:37.832111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:37.832218 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:37.832346 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:37.834770 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.836148 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.839229 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.839958 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.840136 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.845409 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:37.845416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:37.845421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:37.845467 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:37.849462 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.850680 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.850690 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.850697 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.850857 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.851499 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.851506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.851513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.851948 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:37.852128 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:37.853406 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0929 16:01:37.853694 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":266,"threads_started":1}
I20260929 16:01:37.855928 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.828411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:48867)
0929 16:01:37.829381 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:37.829440 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:37.831726 (+ 2286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:37.832669 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:37.832684 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:37.833565 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:37.834543 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.834575 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.836382 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.836390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.839027 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.839045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.840465 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.840471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.840726 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.841699 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:37.841731 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:37.845156 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:37.849630 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.849641 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.849662 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.850523 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.851020 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.851025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.851029 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.851348 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.852194 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:37.852205 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:37.852883 (+ 678us) client_negotiation.cc:769] Sending connection context
0929 16:01:37.853276 (+ 393us) client_negotiation.cc:240] Negotiation successful
0929 16:01:37.854125 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":378,"threads_started":1}
I20260929 16:01:37.858035 21335 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:01:37.858426 21335 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:37.859575 21335 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:01:37.862504 21146 ts_manager.cc:194] Registered new tserver with Master: f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:01:37.864977 21146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:48867
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:01:37.904465 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:38.269811 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:38.270099 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:38.270198 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:38.277279 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:38.277374 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:38.277482 21341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:01:38.286837 21341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:38.288244 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:38.289676 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:38.304523 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.304698 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.304642 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:38.305405 21341 server_base.cc:1034] running on GCE node
I20260929 16:01:38.305977 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:38.306888 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:38.308202 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790697698308144 us; error 55 us; skew 500 ppm
I20260929 16:01:38.308499 21341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:38 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:38.311951 21341 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:01:38.313563 21341 webserver.cc:460] Webserver started at http://127.5.117.2:36467/ using document root <none> and password file <none>
I20260929 16:01:38.314324 21341 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:38.314469 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:38.314821 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:38.318010 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "40d1492dc7174749a875ab3283d7e9a2"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "929461235ef9ab83b4d117aa59f1621f"
server_key_iv: "949e108c23423039f97cb420fcd038e4"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.318840 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "40d1492dc7174749a875ab3283d7e9a2"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "929461235ef9ab83b4d117aa59f1621f"
server_key_iv: "949e108c23423039f97cb420fcd038e4"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.324256 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:01:38.328231 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.329896 21341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:38.330101 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "40d1492dc7174749a875ab3283d7e9a2"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "929461235ef9ab83b4d117aa59f1621f"
server_key_iv: "949e108c23423039f97cb420fcd038e4"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.330305 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:38.346537 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:38.351692 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:38.352020 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:38.353277 21341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:38.355063 21341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:38.355198 21341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.355371 21341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:38.355451 21341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.392956 21341 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:38741
I20260929 16:01:38.392997 21468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:38741 every 8 connection(s)
I20260929 16:01:38.394570 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:38.397076 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21341
I20260929 16:01:38.397353 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:01:38.398047 5588 external_mini_cluster.cc:1428] Setting key b8be4b0974d381a99efb3d8073db4835
Sep 29 16:01:38 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:38.419986 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.397749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:34415 (local address 127.5.117.62:39343)
0929 16:01:38.398300 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.398306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.399678 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 16:01:38.402431 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.402436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.402507 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.402617 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.404925 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.405750 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.408302 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.408756 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.408858 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.413349 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.413353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.413357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.413389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.416908 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.417639 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.417650 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.417654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.417787 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.418227 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.418232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.418234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.418486 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:38.418665 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.419337 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.419573 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
I20260929 16:01:38.420928 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.398445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:34415)
0929 16:01:38.399489 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.399539 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.402085 (+ 2546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.402915 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.402932 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.403797 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:38.404718 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.404746 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.405933 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.405941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.408149 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.408168 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.409000 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.409010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.409383 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.410185 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:38.410219 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:38.413109 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:38.417074 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:38.417082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:38.417096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:38.417503 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:38.417903 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:38.417907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:38.417911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:38.418100 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:38.418632 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:38.418640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:38.418964 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.419225 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.419726 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":452,"threads_started":1}
I20260929 16:01:38.422446 21469 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:01:38.422765 21469 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:38.423511 21469 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:01:38.425155 21146 ts_manager.cc:194] Registered new tserver with Master: 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741)
I20260929 16:01:38.426497 21146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:34415
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:01:38.464231 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:38.850814 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:38.851178 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:38.851327 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:38.859439 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:38.859547 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:38.859747 21475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:01:38.869194 21335 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:01:38.870761 21475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:01:38.872565 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:38.874060 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:38.888620 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.888698 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.889307 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:38.889693 21475 server_base.cc:1034] running on GCE node
I20260929 16:01:38.890309 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:38.891355 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:38.892624 21475 hybrid_clock.cc:648] HybridClock initialized: now 1790697698892564 us; error 47 us; skew 500 ppm
I20260929 16:01:38.893000 21475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:38 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:38.896524 21475 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:01:38.897929 21475 webserver.cc:460] Webserver started at http://127.5.117.3:35365/ using document root <none> and password file <none>
I20260929 16:01:38.898622 21475 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:38.898728 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:38.899097 21475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:38.902263 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "4277236c8caa4af78fd90f83a7a85d0c"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "07c5fdaeb4659df667569c5e083f8ab8"
server_key_iv: "8d00194c9a2b78aa6b108462e945f5fe"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.903012 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "4277236c8caa4af78fd90f83a7a85d0c"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "07c5fdaeb4659df667569c5e083f8ab8"
server_key_iv: "8d00194c9a2b78aa6b108462e945f5fe"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.908447 21475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:01:38.911939 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.913357 21475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:38.913532 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "4277236c8caa4af78fd90f83a7a85d0c"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "07c5fdaeb4659df667569c5e083f8ab8"
server_key_iv: "8d00194c9a2b78aa6b108462e945f5fe"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.913722 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:38.934935 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:38.939535 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:38.939844 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:38.940833 21475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:38.942160 21475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:38.942236 21475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.942392 21475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:38.942441 21475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.977205 21475 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43107
I20260929 16:01:38.977241 21602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43107 every 8 connection(s)
I20260929 16:01:38.978636 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:38.980139 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21475
I20260929 16:01:38.980352 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:01:38.980875 5588 external_mini_cluster.cc:1428] Setting key 2defd7849e4fb7dc4d7cb6742215a092
Sep 29 16:01:38 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:01:39.003201 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.981660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:58613 (local address 127.5.117.62:39343)
0929 16:01:38.982155 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.982160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.983117 (+ 957us) server_negotiation.cc:407] Connection header received
0929 16:01:38.985685 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.985690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.985791 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.985925 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.988167 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.989292 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.991195 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.991667 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.991821 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.996426 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.996435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.996438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.996475 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.999844 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.000674 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.000680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.000684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.000820 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.001294 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.001298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.001300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.001579 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.001759 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.002512 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.002764 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":274,"threads_started":1}
I20260929 16:01:39.004347 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.982155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:58613)
0929 16:01:38.982934 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.983005 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.985390 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.986176 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.986186 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.986746 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:38.987488 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.987531 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.989504 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.989510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.991007 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.991024 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.991961 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.991970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.992296 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.993163 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:38.993188 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:38.996149 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:38.999992 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:39.000002 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:39.000020 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:39.000540 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:39.000948 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:39.000953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:39.000957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:39.001160 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:39.001727 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:39.001735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:39.002104 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.002399 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.002959 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"spinlock_wait_cycles":14080,"thread_start_us":408,"threads_started":1}
I20260929 16:01:39.006196 21603 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:01:39.006767 21603 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:39.007712 21603 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:01:39.009797 21145 ts_manager.cc:194] Registered new tserver with Master: 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107)
I20260929 16:01:39.010700 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:39.011085 21145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:58613
I20260929 16:01:39.032222 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.016624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.5.117.62:39343)
0929 16:01:39.016936 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.016944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.017350 (+ 406us) server_negotiation.cc:407] Connection header received
0929 16:01:39.017528 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.017533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.017616 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.017724 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.019508 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.020307 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.022151 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.022675 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.022844 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.026092 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.026097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.026101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.026140 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.029603 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.030362 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.030366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.030368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.030487 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.030868 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.030871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.030872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.031203 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.031401 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.031714 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.031890 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:01:39.036612 21145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:39.044155 21145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:39.084210 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.062361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.5.117.2:38741)
0929 16:01:39.073580 (+ 11219us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.073586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.073611 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:01:39.073716 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.073721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.073981 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.074280 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:39.075258 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.076934 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.080802 (+ 3868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.082625 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.082860 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.083132 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.083546 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.083746 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11064,"thread_start_us":10869,"threads_started":1}
I20260929 16:01:39.084640 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.062154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:38741 (local address 127.0.0.1:51242)
0929 16:01:39.063155 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:39.063313 (+ 158us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:39.063590 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:39.074282 (+ 10692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:39.074293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:39.074332 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:39.075065 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.075082 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.077081 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.077092 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.080576 (+ 3484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.080592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.082946 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.082952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.083224 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.083285 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.083471 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.083831 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":155,"thread_start_us":384,"threads_started":1}
I20260929 16:01:39.084955 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.063459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36091 (local address 127.0.0.1:52454)
0929 16:01:39.072427 (+ 8968us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:39.072460 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:39.072696 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:39.073884 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:39.073893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:39.073925 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:39.074620 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.074638 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.076902 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.076910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.080953 (+ 4043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.080967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.083282 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.083287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.083468 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.083503 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.083590 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.083747 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8751,"mutex_wait_us":222,"thread_start_us":11331,"threads_started":1}
I20260929 16:01:39.085605 21618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.063767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43107 (local address 127.0.0.1:59032)
0929 16:01:39.075126 (+ 11359us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:39.075142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:39.075296 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:39.076371 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:39.076377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:39.076404 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:39.076772 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.076782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.078389 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.078395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.081060 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.081083 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.083590 (+ 2507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.083597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.084024 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.084086 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.084295 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.084473 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11233,"mutex_wait_us":263,"thread_start_us":8214,"threads_started":1}
I20260929 16:01:39.086489 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.064473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.5.117.1:36091)
0929 16:01:39.071897 (+ 7424us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.071912 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.072884 (+ 972us) server_negotiation.cc:407] Connection header received
0929 16:01:39.073065 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.073076 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.073527 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.073859 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:39.075282 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.076757 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.081171 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.083021 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.085408 (+ 2387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.085619 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.085896 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.086136 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7118,"thread_start_us":6883,"threads_started":1}
I20260929 16:01:39.086511 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.064638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.5.117.3:43107)
0929 16:01:39.073198 (+ 8560us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.073208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.075461 (+ 2253us) server_negotiation.cc:407] Connection header received
0929 16:01:39.075592 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.075599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.075835 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.076027 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:39.077146 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.078242 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.081349 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.083021 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.083171 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.083431 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.085861 (+ 2430us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.086120 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8343,"thread_start_us":8121,"threads_started":1}
I20260929 16:01:39.090261 21404 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1b50f830434be6a0771fdee50174b7 (DEFAULT_TABLE table=test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:39.090263 21538 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1b50f830434be6a0771fdee50174b7 (DEFAULT_TABLE table=test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:39.090973 21270 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1b50f830434be6a0771fdee50174b7 (DEFAULT_TABLE table=test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:39.092042 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1b50f830434be6a0771fdee50174b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:39.092099 21538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1b50f830434be6a0771fdee50174b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:39.092551 21270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1b50f830434be6a0771fdee50174b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:39.106058 21622 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Bootstrap starting.
I20260929 16:01:39.107398 21623 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Bootstrap starting.
I20260929 16:01:39.110332 21622 tablet_bootstrap.cc:654] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:39.110879 21623 tablet_bootstrap.cc:654] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:39.111788 21622 log.cc:824] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:39.112110 21623 log.cc:824] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:39.114931 21624 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Bootstrap starting.
I20260929 16:01:39.115444 21622 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: No bootstrap required, opened a new log
I20260929 16:01:39.115717 21622 ts_tablet_manager.cc:1397] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:01:39.118489 21624 tablet_bootstrap.cc:654] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:39.118824 21623 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: No bootstrap required, opened a new log
I20260929 16:01:39.119089 21623 ts_tablet_manager.cc:1397] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 16:01:39.119702 21624 log.cc:824] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:39.126869 21622 raft_consensus.cc:348] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.127337 21622 raft_consensus.cc:374] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:39.127465 21622 raft_consensus.cc:729] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f529766f1a2b44e4b7a0b1191361d1bd, State: Initialized, Role: FOLLOWER
I20260929 16:01:39.128125 21622 consensus_queue.cc:260] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.133599 21623 raft_consensus.cc:348] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.135133 21623 raft_consensus.cc:374] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:39.135324 21623 raft_consensus.cc:729] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4277236c8caa4af78fd90f83a7a85d0c, State: Initialized, Role: FOLLOWER
I20260929 16:01:39.135831 21622 ts_tablet_manager.cc:1428] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260929 16:01:39.136137 21623 consensus_queue.cc:260] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.138907 21624 tablet_bootstrap.cc:492] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: No bootstrap required, opened a new log
I20260929 16:01:39.139253 21624 ts_tablet_manager.cc:1397] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.016s
I20260929 16:01:39.143373 21603 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:01:39.144438 21623 ts_tablet_manager.cc:1428] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Time spent starting tablet: real 0.025s user 0.021s sys 0.004s
I20260929 16:01:39.149726 21624 raft_consensus.cc:348] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.150110 21624 raft_consensus.cc:374] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:39.150195 21624 raft_consensus.cc:729] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d1492dc7174749a875ab3283d7e9a2, State: Initialized, Role: FOLLOWER
I20260929 16:01:39.150791 21624 consensus_queue.cc:260] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.151966 21469 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:01:39.152588 21624 ts_tablet_manager.cc:1428] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 16:01:39.230512 21604 tablet.cc:2369] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:39.263674 21629 raft_consensus.cc:482] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:39.264211 21629 raft_consensus.cc:504] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.266884 21629 leader_election.cc:288] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741), f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:01:39.274202 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.267468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:44183 (local address 127.5.117.2:38741)
0929 16:01:39.267847 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.267855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.267890 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:01:39.267998 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.268002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.268134 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.268288 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:39.268940 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.269983 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.272302 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.273180 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.273362 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.273472 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.273765 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.273914 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:01:39.274205 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.267393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:38741 (local address 127.5.117.3:44183)
0929 16:01:39.267595 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:39.267620 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:39.267755 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:39.268326 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:39.268332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:39.268394 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:39.268772 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.268791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.270157 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.270164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.272040 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.272086 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.273374 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.273380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.273587 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.273614 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.273739 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.273874 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":25}
I20260929 16:01:39.274966 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.267421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:46157 (local address 127.5.117.1:36091)
0929 16:01:39.267848 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.267855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.268999 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 16:01:39.269280 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.269285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.269388 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.269500 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:39.270460 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.271176 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.273219 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.274075 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.274250 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.274367 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.274651 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.274776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:01:39.275552 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.267393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36091 (local address 127.5.117.3:46157)
0929 16:01:39.268879 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:39.268904 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:39.269047 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:39.269741 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:39.269745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:39.269771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:39.270272 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.270285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.271359 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.271365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.273036 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:39.273048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.274297 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.274303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:39.274473 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.274504 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:01:39.275181 (+ 677us) client_negotiation.cc:240] Negotiation successful
0929 16:01:39.275277 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"mutex_wait_us":462,"thread_start_us":549,"threads_started":1}
I20260929 16:01:39.275455 21424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1b50f830434be6a0771fdee50174b7" candidate_uuid: "4277236c8caa4af78fd90f83a7a85d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d1492dc7174749a875ab3283d7e9a2" is_pre_election: true
I20260929 16:01:39.276021 21424 raft_consensus.cc:2454] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4277236c8caa4af78fd90f83a7a85d0c in term 0.
I20260929 16:01:39.276677 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1b50f830434be6a0771fdee50174b7" candidate_uuid: "4277236c8caa4af78fd90f83a7a85d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" is_pre_election: true
I20260929 16:01:39.276885 21494 leader_election.cc:302] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d1492dc7174749a875ab3283d7e9a2, 4277236c8caa4af78fd90f83a7a85d0c; no voters:
I20260929 16:01:39.277285 21629 raft_consensus.cc:2781] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:39.277280 21290 raft_consensus.cc:2454] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4277236c8caa4af78fd90f83a7a85d0c in term 0.
I20260929 16:01:39.277408 21629 raft_consensus.cc:482] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:39.277521 21629 raft_consensus.cc:3037] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:39.279992 21629 raft_consensus.cc:504] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.280675 21629 leader_election.cc:288] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [CANDIDATE]: Term 1 election: Requested vote from peers 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741), f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:01:39.281328 21424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1b50f830434be6a0771fdee50174b7" candidate_uuid: "4277236c8caa4af78fd90f83a7a85d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40d1492dc7174749a875ab3283d7e9a2"
I20260929 16:01:39.281327 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1b50f830434be6a0771fdee50174b7" candidate_uuid: "4277236c8caa4af78fd90f83a7a85d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
I20260929 16:01:39.281553 21290 raft_consensus.cc:3037] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:39.281553 21424 raft_consensus.cc:3037] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:39.285063 21424 raft_consensus.cc:2454] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4277236c8caa4af78fd90f83a7a85d0c in term 1.
I20260929 16:01:39.285063 21290 raft_consensus.cc:2454] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4277236c8caa4af78fd90f83a7a85d0c in term 1.
I20260929 16:01:39.285545 21491 leader_election.cc:302] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4277236c8caa4af78fd90f83a7a85d0c, f529766f1a2b44e4b7a0b1191361d1bd; no voters:
I20260929 16:01:39.285987 21629 raft_consensus.cc:2781] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:39.286520 21629 raft_consensus.cc:686] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 LEADER]: Becoming Leader. State: Replica: 4277236c8caa4af78fd90f83a7a85d0c, State: Running, Role: LEADER
I20260929 16:01:39.287109 21629 consensus_queue.cc:237] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } }
I20260929 16:01:39.293180 21145 catalog_manager.cc:5495] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3). New cstate: current_term: 1 leader_uuid: "4277236c8caa4af78fd90f83a7a85d0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } health_report { overall_health: UNKNOWN } } }
W20260929 16:01:39.327854 21336 tablet.cc:2369] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:39.333209 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.325207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.5.117.3:43107)
0929 16:01:39.325505 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.325510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.325569 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:01:39.325988 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.325992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.326083 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.326266 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:39.326804 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.328390 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.330191 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.330658 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.330773 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.331492 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:39.332463 (+ 971us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:39.332608 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.332846 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.332944 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260929 16:01:39.342921 21290 raft_consensus.cc:1261] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Refusing update from remote peer 4277236c8caa4af78fd90f83a7a85d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:39.343393 21424 raft_consensus.cc:1261] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Refusing update from remote peer 4277236c8caa4af78fd90f83a7a85d0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:39.344132 21632 consensus_queue.cc:1035] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:39.344643 21629 consensus_queue.cc:1035] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:39.356547 21635 mvcc.cc:205] Tried to move back new op lower bound from 7334697776497811456 to 7334697776285663232. Current Snapshot: MvccSnapshot[applied={T|T < 7334697776497811456}]
I20260929 16:01:39.371925 21145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:39.372247 21145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:39.376143 21145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Sending DeleteTablet for 3 replicas of tablet 9f1b50f830434be6a0771fdee50174b7
I20260929 16:01:39.377328 21270 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1b50f830434be6a0771fdee50174b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52454
I20260929 16:01:39.377447 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21207
I20260929 16:01:39.377568 21404 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1b50f830434be6a0771fdee50174b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:51242
I20260929 16:01:39.378062 21538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1b50f830434be6a0771fdee50174b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:39 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:59032
I20260929 16:01:39.384653 21645 tablet_replica.cc:330] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: stopping tablet replica
I20260929 16:01:39.385210 21645 raft_consensus.cc:2229] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:39.385819 21645 raft_consensus.cc:2258] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:39.393330 21646 tablet_replica.cc:330] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: stopping tablet replica
I20260929 16:01:39.393662 21647 tablet_replica.cc:330] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: stopping tablet replica
I20260929 16:01:39.394088 21647 raft_consensus.cc:2229] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:39.394506 21646 raft_consensus.cc:2229] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:39.395007 21647 raft_consensus.cc:2258] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:39.395097 21646 raft_consensus.cc:2258] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:39.396374 21645 ts_tablet_manager.cc:1905] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:39.397230 21647 ts_tablet_manager.cc:1905] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:39.404821 21646 ts_tablet_manager.cc:1905] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:39.404987 21647 ts_tablet_manager.cc:1918] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:39.405179 21647 log.cc:1196] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/9f1b50f830434be6a0771fdee50174b7
I20260929 16:01:39.405766 21647 ts_tablet_manager.cc:1939] T 9f1b50f830434be6a0771fdee50174b7 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting consensus metadata
I20260929 16:01:39.407701 21133 catalog_manager.cc:4841] TS 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107): tablet 9f1b50f830434be6a0771fdee50174b7 (table test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]) successfully deleted
Sep 29 16:01:39 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:01:39.412885 21646 ts_tablet_manager.cc:1918] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:39.413046 21646 log.cc:1196] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/9f1b50f830434be6a0771fdee50174b7
I20260929 16:01:39.413602 21646 ts_tablet_manager.cc:1939] T 9f1b50f830434be6a0771fdee50174b7 P 40d1492dc7174749a875ab3283d7e9a2: Deleting consensus metadata
I20260929 16:01:39.415299 21133 catalog_manager.cc:4841] TS 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741): tablet 9f1b50f830434be6a0771fdee50174b7 (table test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]) successfully deleted
I20260929 16:01:39.416971 21645 ts_tablet_manager.cc:1918] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:39.417100 21645 log.cc:1196] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/9f1b50f830434be6a0771fdee50174b7
I20260929 16:01:39.417547 21645 ts_tablet_manager.cc:1939] T 9f1b50f830434be6a0771fdee50174b7 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting consensus metadata
I20260929 16:01:39.419342 21130 catalog_manager.cc:4841] TS f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091): tablet 9f1b50f830434be6a0771fdee50174b7 (table test-table [id=b53eb10c452f44cc9a9b6e0928b3e9e4]) successfully deleted
I20260929 16:01:39.420970 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.378776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.5.117.1:36091)
0929 16:01:39.379012 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.379022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.395481 (+ 16459us) server_negotiation.cc:407] Connection header received
0929 16:01:39.395572 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.395578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.395672 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.395804 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.398155 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.399387 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.401494 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.402569 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.402680 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.413044 (+ 10364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.413076 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.413092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.413161 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.417157 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.418163 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.418174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.418177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.418348 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.418934 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.418939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.418942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.419425 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.419738 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.420357 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.420592 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:01:39.421778 21330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.526101 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21207
I20260929 16:01:39.550181 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21341
Sep 29 16:01:39 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:01:39.568558 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.551346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.5.117.2:38741)
0929 16:01:39.551556 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.551563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.551807 (+ 244us) server_negotiation.cc:407] Connection header received
0929 16:01:39.552238 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.552243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.552359 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.552499 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.553968 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.554779 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.556314 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.556736 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.556880 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.561914 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.561933 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.561949 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.561983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.565503 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.566333 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.566342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.566344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.566479 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.566859 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.566865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.566868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.567272 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.567612 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.567915 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.568198 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:01:39.569362 21464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.698695 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21341
I20260929 16:01:39.720448 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21475
Sep 29 16:01:39 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:01:39.741024 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.721377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.5.117.3:43107)
0929 16:01:39.721623 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.721632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.721726 (+ 94us) server_negotiation.cc:407] Connection header received
0929 16:01:39.721969 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.721973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.722048 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.722166 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.723655 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.724670 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.726103 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.726630 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.726785 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.733013 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.733035 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.733060 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.733120 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.737005 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.738059 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.738068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.738070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.738263 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.738696 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.738704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.738708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.739213 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.739620 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.739944 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.740664 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:01:39.741943 21598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.886893 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21475
I20260929 16:01:39.910427 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21114
I20260929 16:01:39.911729 21176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:40.041116 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21114
I20260929 16:02:00.074311 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--webserver_interface=127.5.117.62
--webserver_port=46335
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:00.444896 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:00.445195 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:00.445376 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:00.452533 21653 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:02:00.452641 21653 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:02:00.452740 21653 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:02:00.452837 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:00.452939 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:00.453070 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:00.453150 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:00.453240 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:00.463307 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=46335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:00.464920 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:00.466403 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:00.493613 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.493592 21660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.494336 21662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:00.494591 21653 server_base.cc:1034] running on GCE node
I20260929 16:02:00.495622 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:00.497578 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:00.498852 21653 hybrid_clock.cc:648] HybridClock initialized: now 1790697720498783 us; error 39 us; skew 500 ppm
I20260929 16:02:00.499289 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:00 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697720, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:00.503283 21653 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:02:00.505234 21653 webserver.cc:460] Webserver started at http://127.5.117.62:46335/ using document root <none> and password file <none>
I20260929 16:02:00.506131 21653 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:00.506297 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:00.512795 21653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:02:00.516126 21669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.517522 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:02:00.517732 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "fd82de88e6dd4e6393aa16e939b49bcb"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "8f4901ae47d34c33484de88f6fc55760"
server_key_iv: "d5aca71fbc24bd36935848e00ed19138"
server_key_version: "encryptionkey@0"
I20260929 16:02:00.518981 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:00.543215 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:00.547686 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:00.548019 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:00.569206 21653 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39343
I20260929 16:02:00.569231 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39343 every 8 connection(s)
I20260929 16:02:00.570887 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:00.576683 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21653
I20260929 16:02:00.577859 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:36091
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=36427
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:02:00.579732 21721 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:02:00.585317 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap starting.
I20260929 16:02:00.609581 21721 log.cc:824] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:00.623500 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:02:00.624205 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap complete.
I20260929 16:02:00.634235 21721 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:00.635085 21721 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Initialized, Role: FOLLOWER
I20260929 16:02:00.635876 21721 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:00.636170 21721 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:00.636322 21721 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:00.636539 21721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:02:00.640681 21721 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:00.641278 21721 leader_election.cc:302] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd82de88e6dd4e6393aa16e939b49bcb; no voters:
I20260929 16:02:00.641995 21721 leader_election.cc:288] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:02:00.642153 21725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:02:00.643440 21725 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 LEADER]: Becoming Leader. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Running, Role: LEADER
I20260929 16:02:00.644134 21725 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:00.644590 21721 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:00.646934 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd82de88e6dd4e6393aa16e939b49bcb. Latest consensus state: current_term: 2 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:02:00.647306 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:00.647027 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:02:00.647464 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:00.648512 21731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:00.665365 21731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:00.667390 21731 catalog_manager.cc:1259] Loaded cluster ID: 56f52f1a96c54476bb7c2957200260f5
I20260929 16:02:00.667608 21731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:00.673075 21731 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:00.675444 21731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Loaded TSK: 0
I20260929 16:02:00.708595 21742 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:02:00.723996 21731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Generated new TSK 1
I20260929 16:02:00.724572 21731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:02:00.995389 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:00.995745 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:00.995936 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:01.003237 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:01.003383 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:01.003580 21723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:01.013301 21723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:36091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=36427
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:01.014827 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:01.016525 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:01.031757 21749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:01.032368 21751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:01.032838 21723 server_base.cc:1034] running on GCE node
W20260929 16:02:01.032976 21748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:01.033707 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:01.034849 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:01.036127 21723 hybrid_clock.cc:648] HybridClock initialized: now 1790697721036069 us; error 45 us; skew 500 ppm
I20260929 16:02:01.036474 21723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:01 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:01.040176 21723 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:02:01.041858 21723 webserver.cc:460] Webserver started at http://127.5.117.1:36427/ using document root <none> and password file <none>
I20260929 16:02:01.042639 21723 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:01.042816 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:01.049196 21723 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:02:01.052549 21758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.054039 21723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:01.054232 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "940d9d1e4884205c6ff808c3cfdfc1c6"
server_key_iv: "1524151bc912abcf781b48acefd1f5bf"
server_key_version: "encryptionkey@0"
I20260929 16:02:01.055310 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:01.092430 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:01.096607 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:01.096882 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:01.097935 21723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:01.099321 21723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:01.099455 21723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.099592 21723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:01.099684 21723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.135432 21723 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36091
I20260929 16:02:01.135507 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36091 every 8 connection(s)
I20260929 16:02:01.137199 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:01.142410 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21723
I20260929 16:02:01.143563 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:38741
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=36467
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:02:01 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:01.167253 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.141344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:48485 (local address 127.5.117.62:39343)
0929 16:02:01.142046 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:01.142059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:01.144799 (+ 2740us) server_negotiation.cc:407] Connection header received
0929 16:02:01.147189 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:01.147202 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:01.147526 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:01.147910 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:01.150569 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.151841 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.153486 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.154134 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.154338 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:01.159130 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:01.159138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:01.159151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:01.159189 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:01.162386 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:01.163390 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:01.163399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:01.163405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:01.163682 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:01.164254 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:01.164257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:01.164259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:01.164831 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:01.164985 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:01.165942 (+ 957us) server_negotiation.cc:299] Negotiation successful
0929 16:02:01.166257 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":336,"threads_started":1}
I20260929 16:02:01.167603 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.141501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:48485)
0929 16:02:01.144612 (+ 3111us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.144657 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:01.146607 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:01.149204 (+ 2597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:01.149211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:01.149759 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:01.150340 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:01.150358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.152012 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.152022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:01.153325 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:01.153338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.154372 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.154379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:01.154705 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:01.155597 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:01.155635 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:01.158826 (+ 3191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:01.162528 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:01.162539 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:01.162555 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:01.163224 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:01.163842 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:01.163850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:01.163857 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:01.164138 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:01.164965 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:01.164975 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:01.165430 (+ 455us) client_negotiation.cc:769] Sending connection context
0929 16:02:01.165732 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 16:02:01.166330 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":2623,"threads_started":1}
I20260929 16:02:01.170960 21871 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:02:01.171356 21871 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:01.172500 21871 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:02:01.175434 21686 ts_manager.cc:194] Registered new tserver with Master: f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:02:01.177876 21686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:48485
W20260929 16:02:01.526971 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:01.527345 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:01.527451 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:01.535526 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:01.535691 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:01.535826 21875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:01.547565 21875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:38741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=36467
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:01.549360 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:01.550927 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:01.566583 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:01.566535 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:01.566619 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:01.567761 21875 server_base.cc:1034] running on GCE node
I20260929 16:02:01.568629 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:01.570011 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:01.571267 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790697721571199 us; error 54 us; skew 500 ppm
I20260929 16:02:01.571671 21875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:01 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:01.575950 21875 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:02:01.587657 21875 webserver.cc:460] Webserver started at http://127.5.117.2:36467/ using document root <none> and password file <none>
I20260929 16:02:01.588687 21875 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:01.588876 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:01.595644 21875 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:02:01.598924 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.600521 21875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:02:01.600739 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "40d1492dc7174749a875ab3283d7e9a2"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "929461235ef9ab83b4d117aa59f1621f"
server_key_iv: "949e108c23423039f97cb420fcd038e4"
server_key_version: "encryptionkey@0"
I20260929 16:02:01.601933 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:01.621826 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:01.626855 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:01.627185 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:01.628441 21875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:01.629993 21875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:01.630127 21875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.630290 21875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:01.630385 21875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.669483 21875 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:38741
I20260929 16:02:01.669780 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:38741 every 8 connection(s)
I20260929 16:02:01.671454 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:01.680038 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 21875
I20260929 16:02:01.681063 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:43107
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=35365
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:02:01 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697721, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:01.698297 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.674875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:56577 (local address 127.5.117.62:39343)
0929 16:02:01.675517 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:01.675523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:01.676508 (+ 985us) server_negotiation.cc:407] Connection header received
0929 16:02:01.678478 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:01.678484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:01.678584 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:01.678727 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:01.680510 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.681745 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.683486 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.684320 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.684477 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:01.688989 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:01.689000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:01.689008 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:01.689066 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:01.693492 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:01.694746 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:01.694751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:01.694754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:01.695182 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:01.695847 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:01.695852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:01.695855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:01.696341 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:01.696561 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:01.697576 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 16:02:01.697866 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":302,"threads_started":1}
I20260929 16:02:01.699636 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.675354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:56577)
0929 16:02:01.676339 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.676385 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:01.678165 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:01.678967 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:01.678974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:01.679530 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:01.680286 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:01.680313 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.681954 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.681959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:01.683320 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:01.683336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:01.684648 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:01.684658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:01.684889 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:01.685708 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:01.685732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:01.688703 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:01.693719 (+ 5016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:01.693745 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:01.693777 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:01.694539 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:01.695380 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:01.695388 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:01.695394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:01.695671 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:01.696521 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:01.696531 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:01.697109 (+ 578us) client_negotiation.cc:769] Sending connection context
0929 16:02:01.697454 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 16:02:01.698145 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":380,"threads_started":1}
I20260929 16:02:01.701575 22003 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:02:01.701977 22003 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:01.702890 22003 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:02:01.704742 21686 ts_manager.cc:194] Registered new tserver with Master: 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741)
I20260929 16:02:01.705690 21686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:56577
W20260929 16:02:02.070827 22007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:02.071171 22007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:02.071352 22007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:02.078610 22007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:02.078763 22007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:02.078933 22007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:02.088526 22007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:43107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=35365
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:02.090009 22007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:02.091572 22007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:02.107419 22012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:02.107458 22013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:02.107605 22015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:02.108335 22007 server_base.cc:1034] running on GCE node
I20260929 16:02:02.109197 22007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:02.110625 22007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:02.111929 22007 hybrid_clock.cc:648] HybridClock initialized: now 1790697722111872 us; error 57 us; skew 500 ppm
I20260929 16:02:02.112442 22007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:02 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:02.116878 22007 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:02:02.118789 22007 webserver.cc:460] Webserver started at http://127.5.117.3:35365/ using document root <none> and password file <none>
I20260929 16:02:02.119697 22007 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:02.119889 22007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:02.126948 22007 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:02.130447 22022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:02.132226 22007 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:02:02.132447 22007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "4277236c8caa4af78fd90f83a7a85d0c"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "07c5fdaeb4659df667569c5e083f8ab8"
server_key_iv: "8d00194c9a2b78aa6b108462e945f5fe"
server_key_version: "encryptionkey@0"
I20260929 16:02:02.133677 22007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:02.152199 22007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:02.157080 22007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:02.157397 22007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:02.158742 22007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:02.160454 22007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:02.160570 22007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:02.160651 22007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:02.160708 22007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:02.182140 21871 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:02:02.199512 22007 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43107
I20260929 16:02:02.199570 22134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43107 every 8 connection(s)
I20260929 16:02:02.201511 22007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:02.207957 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22007
Sep 29 16:02:02 dist-test-slave-b69v krb5kdc[21101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697722, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:02.228837 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.204438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:52289 (local address 127.5.117.62:39343)
0929 16:02:02.204945 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.204950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.206085 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 16:02:02.208286 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.208293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.208405 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.208547 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:02.210427 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.211226 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.213084 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.213864 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.214047 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.220146 (+ 6099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:02.220154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:02.220160 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:02.220210 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:02.224389 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.225603 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.225607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.225610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.225744 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.226358 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.226362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.226364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.226731 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:02.226861 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.228079 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.228361 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260929 16:02:02.230398 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.205066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:52289)
0929 16:02:02.205907 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.205974 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.207931 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.208811 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.208824 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.209630 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:02.210251 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.210268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.211395 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.211401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.212923 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.212937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.214378 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.214388 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.214741 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.215783 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:02.215825 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:02.219815 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:02.224583 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:02.224597 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:02.224625 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:02.225434 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:02.225871 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:02.225881 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:02.225883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:02.226209 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:02.226875 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:02.226888 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:02.227517 (+ 629us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.227878 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.228525 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":296,"threads_started":1}
I20260929 16:02:02.231145 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.208976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.5.117.62:39343)
0929 16:02:02.209531 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.209538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.212663 (+ 3125us) server_negotiation.cc:407] Connection header received
0929 16:02:02.212811 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.212816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.212969 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.213085 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:02.215334 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.216874 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.218906 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.219598 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.219765 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.223974 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:02.223980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:02.223983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:02.224040 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:02.227502 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.228570 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.228575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.228578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.228730 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.229383 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.229389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.229394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.229774 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:02.230073 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.230431 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.230699 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260929 16:02:02.232666 22135 heartbeater.cc:347] Connected to a master server at 127.5.117.62:39343
I20260929 16:02:02.233325 22135 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:02.234613 22135 heartbeater.cc:511] Master 127.5.117.62:39343 requested a full tablet report, sending...
I20260929 16:02:02.236565 21685 ts_manager.cc:194] Registered new tserver with Master: 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107)
I20260929 16:02:02.238152 21685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:52289
I20260929 16:02:02.247448 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:02.255357 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.248699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.5.117.62:39343)
0929 16:02:02.248916 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.248922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.249505 (+ 583us) server_negotiation.cc:407] Connection header received
0929 16:02:02.249701 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.249705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.249913 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.250068 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:02.250583 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.251385 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.253103 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.253565 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.253666 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.254218 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:02.254502 (+ 284us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:02:02.255039 (+ 537us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54690: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 16:02:02.255587 22139 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54690: token expired
W20260929 16:02:02.256194 21608 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.117.62:39343)
I20260929 16:02:02.272413 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.256987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.5.117.62:39343)
0929 16:02:02.257200 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.257205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.257222 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:02:02.257366 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.257369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.257438 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.257549 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:02.259287 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.260261 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.261939 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.262407 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.262538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.265957 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:02.265965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:02.265970 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:02.266017 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:02.269074 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.269906 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.269910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.269913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.270072 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.270442 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.270446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.270448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.270832 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:02.271059 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.271806 (+ 747us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.272024 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:02:02.276911 21685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:02.284852 21685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:02.312904 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.299437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.5.117.3:43107)
0929 16:02:02.301423 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.301434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.301474 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:02:02.301589 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.301597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.302192 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.303045 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:02.303381 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.305093 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.309142 (+ 4049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.311580 (+ 2438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.312074 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.312294 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.312392 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.312554 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":1633,"threads_started":1}
I20260929 16:02:02.312952 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.299078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43107 (local address 127.0.0.1:42088)
0929 16:02:02.300329 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.300386 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.300617 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.302444 (+ 1827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.302451 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.302483 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:02.303048 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.303064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.306227 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.306233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.308877 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.308893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.311837 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.311843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.312034 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.312117 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.312243 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.312457 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":20,"thread_start_us":1728,"threads_started":1}
I20260929 16:02:02.314049 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.299856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.5.117.1:36091)
0929 16:02:02.301862 (+ 2006us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.301873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.303656 (+ 1783us) server_negotiation.cc:407] Connection header received
0929 16:02:02.303993 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.304002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.304259 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.304438 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:02.305334 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.306992 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.309570 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.311098 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.311214 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.311456 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.313481 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.313676 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805,"thread_start_us":1629,"threads_started":1}
I20260929 16:02:02.314908 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.299192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36091 (local address 127.0.0.1:59210)
0929 16:02:02.303531 (+ 4339us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.303551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.303771 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.304747 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.304751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.304777 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:02.305133 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.305148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.307181 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.307186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.309365 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.309380 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.311719 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.311728 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.311960 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.313012 (+ 1052us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.314333 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.314470 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4188,"mutex_wait_us":3,"thread_start_us":4027,"threads_started":1}
I20260929 16:02:02.317199 22070 tablet_service.cc:1467] Processing CreateTablet for tablet 57febfb335334361a8bc26123948dd68 (DEFAULT_TABLE table=test-table [id=03bc3d2e2c844b0284ece99e3de715a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:02.318806 22070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57febfb335334361a8bc26123948dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:02.318499 21806 tablet_service.cc:1467] Processing CreateTablet for tablet 57febfb335334361a8bc26123948dd68 (DEFAULT_TABLE table=test-table [id=03bc3d2e2c844b0284ece99e3de715a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:02.319803 21806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57febfb335334361a8bc26123948dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:02.320259 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.302227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:38741 (local address 127.0.0.1:60374)
0929 16:02:02.311887 (+ 9660us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.311909 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.312089 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.312703 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.312707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.312732 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:02.313223 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.313295 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.315550 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.315557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.317567 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.317582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.319426 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.319433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.319615 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.319653 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.319755 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.319896 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9537,"thread_start_us":9344,"threads_started":1}
I20260929 16:02:02.321153 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.302573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.5.117.2:38741)
0929 16:02:02.311517 (+ 8944us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.311529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.312025 (+ 496us) server_negotiation.cc:407] Connection header received
0929 16:02:02.312279 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.312286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.312496 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.312723 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:02.313488 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.315414 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.317748 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.319195 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.319324 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.319509 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.319819 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.320784 (+ 965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8771,"thread_start_us":8610,"threads_started":1}
I20260929 16:02:02.324532 21938 tablet_service.cc:1467] Processing CreateTablet for tablet 57febfb335334361a8bc26123948dd68 (DEFAULT_TABLE table=test-table [id=03bc3d2e2c844b0284ece99e3de715a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:02.325942 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57febfb335334361a8bc26123948dd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:02.332226 22152 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Bootstrap starting.
I20260929 16:02:02.335520 22152 tablet_bootstrap.cc:654] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:02.337078 22152 log.cc:824] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:02.337335 22153 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Bootstrap starting.
I20260929 16:02:02.341081 22152 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: No bootstrap required, opened a new log
I20260929 16:02:02.341357 22155 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Bootstrap starting.
I20260929 16:02:02.341487 22153 tablet_bootstrap.cc:654] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:02.341454 22152 ts_tablet_manager.cc:1397] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:02:02.342798 22153 log.cc:824] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:02.344750 22155 tablet_bootstrap.cc:654] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:02.345736 22155 log.cc:824] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:02.345932 22153 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: No bootstrap required, opened a new log
I20260929 16:02:02.346271 22153 ts_tablet_manager.cc:1397] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:02:02.349973 22155 tablet_bootstrap.cc:492] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: No bootstrap required, opened a new log
I20260929 16:02:02.350229 22155 ts_tablet_manager.cc:1397] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 16:02:02.351977 22152 raft_consensus.cc:348] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.352460 22152 raft_consensus.cc:374] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:02.352612 22152 raft_consensus.cc:729] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4277236c8caa4af78fd90f83a7a85d0c, State: Initialized, Role: FOLLOWER
I20260929 16:02:02.353281 22152 consensus_queue.cc:260] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.354440 22135 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:02:02.356443 22152 ts_tablet_manager.cc:1428] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 16:02:02.356493 22153 raft_consensus.cc:348] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.356895 22153 raft_consensus.cc:374] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:02.357002 22153 raft_consensus.cc:729] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f529766f1a2b44e4b7a0b1191361d1bd, State: Initialized, Role: FOLLOWER
I20260929 16:02:02.357549 22153 consensus_queue.cc:260] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.358603 22155 raft_consensus.cc:348] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.359138 22155 raft_consensus.cc:374] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:02.359284 22155 raft_consensus.cc:729] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d1492dc7174749a875ab3283d7e9a2, State: Initialized, Role: FOLLOWER
I20260929 16:02:02.359303 22153 ts_tablet_manager.cc:1428] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:02:02.360081 22155 consensus_queue.cc:260] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.361585 22003 heartbeater.cc:503] Master 127.5.117.62:39343 was elected leader, sending a full tablet report...
I20260929 16:02:02.362200 22155 ts_tablet_manager.cc:1428] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
W20260929 16:02:02.391090 21872 tablet.cc:2369] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:02.423933 22004 tablet.cc:2369] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:02.453649 22136 tablet.cc:2369] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:02.592211 22160 raft_consensus.cc:482] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:02.592703 22160 raft_consensus.cc:504] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.595151 22160 leader_election.cc:288] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107), f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:02:02.603686 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.595735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:55291 (local address 127.5.117.1:36091)
0929 16:02:02.595977 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.595986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.596992 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 16:02:02.597185 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.597193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.597279 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.597353 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:02.598062 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.599055 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.601445 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.602757 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.602893 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.602992 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.603308 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.603436 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:02:02.603673 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.595529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:44147 (local address 127.5.117.3:43107)
0929 16:02:02.595834 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.595840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.596565 (+ 725us) server_negotiation.cc:407] Connection header received
0929 16:02:02.596832 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.596836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.596901 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.597013 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:02.597624 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.598687 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.601281 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.602600 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.602734 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.602857 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.603254 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.603390 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 16:02:02.603852 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.595561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43107 (local address 127.5.117.2:44147)
0929 16:02:02.596442 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.596458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.596654 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.597053 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.597060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.597090 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:02.597461 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.597474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.598823 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.598827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.601115 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.601129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.602855 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.602862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.603053 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.603098 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.603204 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.603350 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":999,"threads_started":1}
I20260929 16:02:02.604439 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.595591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36091 (local address 127.5.117.2:55291)
0929 16:02:02.596868 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.596882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:02.597020 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:02.597413 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:02.597422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:02.597452 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:02.597881 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.597890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.599178 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.599183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.601290 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:02.601300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.602968 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.602973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:02.603145 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.603188 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:02:02.603266 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:02:02.603361 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":500,"threads_started":1}
I20260929 16:02:02.604923 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57febfb335334361a8bc26123948dd68" candidate_uuid: "40d1492dc7174749a875ab3283d7e9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4277236c8caa4af78fd90f83a7a85d0c" is_pre_election: true
I20260929 16:02:02.605569 22090 raft_consensus.cc:2454] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d1492dc7174749a875ab3283d7e9a2 in term 0.
I20260929 16:02:02.605612 21826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57febfb335334361a8bc26123948dd68" candidate_uuid: "40d1492dc7174749a875ab3283d7e9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" is_pre_election: true
I20260929 16:02:02.606243 21826 raft_consensus.cc:2454] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d1492dc7174749a875ab3283d7e9a2 in term 0.
I20260929 16:02:02.606341 21894 leader_election.cc:302] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d1492dc7174749a875ab3283d7e9a2, 4277236c8caa4af78fd90f83a7a85d0c; no voters:
I20260929 16:02:02.606871 22160 raft_consensus.cc:2781] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:02.607043 22160 raft_consensus.cc:482] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:02.607167 22160 raft_consensus.cc:3037] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:02.609658 22160 raft_consensus.cc:504] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.610226 22160 leader_election.cc:288] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [CANDIDATE]: Term 1 election: Requested vote from peers 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107), f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091)
I20260929 16:02:02.610670 21826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57febfb335334361a8bc26123948dd68" candidate_uuid: "40d1492dc7174749a875ab3283d7e9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
I20260929 16:02:02.610670 22090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57febfb335334361a8bc26123948dd68" candidate_uuid: "40d1492dc7174749a875ab3283d7e9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4277236c8caa4af78fd90f83a7a85d0c"
I20260929 16:02:02.610885 22090 raft_consensus.cc:3037] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:02.610885 21826 raft_consensus.cc:3037] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:02.614293 21826 raft_consensus.cc:2454] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d1492dc7174749a875ab3283d7e9a2 in term 1.
I20260929 16:02:02.614293 22090 raft_consensus.cc:2454] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d1492dc7174749a875ab3283d7e9a2 in term 1.
I20260929 16:02:02.614704 21894 leader_election.cc:302] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d1492dc7174749a875ab3283d7e9a2, 4277236c8caa4af78fd90f83a7a85d0c; no voters:
I20260929 16:02:02.615067 22160 raft_consensus.cc:2781] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:02.615572 22160 raft_consensus.cc:686] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 40d1492dc7174749a875ab3283d7e9a2, State: Running, Role: LEADER
I20260929 16:02:02.616139 22160 consensus_queue.cc:237] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } }
I20260929 16:02:02.620951 21686 catalog_manager.cc:5495] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "40d1492dc7174749a875ab3283d7e9a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40d1492dc7174749a875ab3283d7e9a2" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38741 } health_report { overall_health: HEALTHY } } }
I20260929 16:02:02.681000 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.675132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.5.117.2:38741)
0929 16:02:02.675363 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.675372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.675442 (+ 70us) server_negotiation.cc:407] Connection header received
0929 16:02:02.675708 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.675713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.675799 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.675963 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:02.676421 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.677232 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.678705 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.679169 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.679307 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.679797 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:02.680335 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:02.680500 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.680611 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.680698 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:02:02.690302 22090 raft_consensus.cc:1261] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Refusing update from remote peer 40d1492dc7174749a875ab3283d7e9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:02.690613 21826 raft_consensus.cc:1261] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Refusing update from remote peer 40d1492dc7174749a875ab3283d7e9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:02.691371 22160 consensus_queue.cc:1035] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4277236c8caa4af78fd90f83a7a85d0c" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:02.691811 22163 consensus_queue.cc:1035] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f529766f1a2b44e4b7a0b1191361d1bd" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:02.708794 22167 mvcc.cc:205] Tried to move back new op lower bound from 7334697872128385024 to 7334697871839854592. Current Snapshot: MvccSnapshot[applied={T|T < 7334697872128385024}]
I20260929 16:02:02.730157 21684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:02.730535 21684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:02.735142 21684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Sending DeleteTablet for 3 replicas of tablet 57febfb335334361a8bc26123948dd68
I20260929 16:02:02.736755 21806 tablet_service.cc:1514] Processing DeleteTablet for tablet 57febfb335334361a8bc26123948dd68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:59210
I20260929 16:02:02.736768 22070 tablet_service.cc:1514] Processing DeleteTablet for tablet 57febfb335334361a8bc26123948dd68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42088
I20260929 16:02:02.737326 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21723
I20260929 16:02:02.737708 21938 tablet_service.cc:1514] Processing DeleteTablet for tablet 57febfb335334361a8bc26123948dd68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:02 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:60374
I20260929 16:02:02.746701 22178 tablet_replica.cc:330] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: stopping tablet replica
I20260929 16:02:02.747426 22178 raft_consensus.cc:2229] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:02.747550 22177 tablet_replica.cc:330] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: stopping tablet replica
I20260929 16:02:02.748531 22177 raft_consensus.cc:2229] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:02.748729 22178 raft_consensus.cc:2258] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:02.749248 22176 tablet_replica.cc:330] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: stopping tablet replica
I20260929 16:02:02.749609 22177 raft_consensus.cc:2258] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:02.749861 22176 raft_consensus.cc:2229] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:02.750499 22176 raft_consensus.cc:2258] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:02.751955 22177 ts_tablet_manager.cc:1905] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:02.751972 22178 ts_tablet_manager.cc:1905] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:02.760885 22176 ts_tablet_manager.cc:1905] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:02.761152 22178 ts_tablet_manager.cc:1918] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:02.761369 22178 log.cc:1196] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/57febfb335334361a8bc26123948dd68
I20260929 16:02:02.762907 22178 ts_tablet_manager.cc:1939] T 57febfb335334361a8bc26123948dd68 P 40d1492dc7174749a875ab3283d7e9a2: Deleting consensus metadata
I20260929 16:02:02.763732 22177 ts_tablet_manager.cc:1918] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:02.763963 22177 log.cc:1196] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/57febfb335334361a8bc26123948dd68
I20260929 16:02:02.764688 22177 ts_tablet_manager.cc:1939] T 57febfb335334361a8bc26123948dd68 P f529766f1a2b44e4b7a0b1191361d1bd: Deleting consensus metadata
I20260929 16:02:02.765756 21673 catalog_manager.cc:4841] TS 40d1492dc7174749a875ab3283d7e9a2 (127.5.117.2:38741): tablet 57febfb335334361a8bc26123948dd68 (table test-table [id=03bc3d2e2c844b0284ece99e3de715a6]) successfully deleted
I20260929 16:02:02.767529 21670 catalog_manager.cc:4841] TS f529766f1a2b44e4b7a0b1191361d1bd (127.5.117.1:36091): tablet 57febfb335334361a8bc26123948dd68 (table test-table [id=03bc3d2e2c844b0284ece99e3de715a6]) successfully deleted
I20260929 16:02:02.770172 22176 ts_tablet_manager.cc:1918] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:02.770386 22176 log.cc:1196] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/57febfb335334361a8bc26123948dd68
I20260929 16:02:02.770829 22176 ts_tablet_manager.cc:1939] T 57febfb335334361a8bc26123948dd68 P 4277236c8caa4af78fd90f83a7a85d0c: Deleting consensus metadata
I20260929 16:02:02.773222 21673 catalog_manager.cc:4841] TS 4277236c8caa4af78fd90f83a7a85d0c (127.5.117.3:43107): tablet 57febfb335334361a8bc26123948dd68 (table test-table [id=03bc3d2e2c844b0284ece99e3de715a6]) successfully deleted
I20260929 16:02:02.774127 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.738832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.5.117.1:36091)
0929 16:02:02.739110 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.739121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.747616 (+ 8495us) server_negotiation.cc:407] Connection header received
0929 16:02:02.748156 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.748161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.748280 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.748371 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:02.753506 (+ 5135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.754821 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.757161 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.757857 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.757973 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.763422 (+ 5449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:02.763452 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:02.763490 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:02.763554 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:02.769324 (+ 5770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.770495 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.770509 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.770513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.770711 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.771276 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.771283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.771287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.771999 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:02.772397 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.773437 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.773743 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 16:02:02.774902 21866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:02.874658 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21723
I20260929 16:02:02.896667 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21875
I20260929 16:02:02.912774 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.897596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.5.117.2:38741)
0929 16:02:02.897813 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:02.897820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:02.897932 (+ 112us) server_negotiation.cc:407] Connection header received
0929 16:02:02.898175 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:02.898182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:02.898271 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:02.898398 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:02.899912 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.900892 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.902240 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:02.902724 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:02.902878 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:02.905873 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:02.905885 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:02.905904 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:02.905946 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:02.909531 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.910278 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.910286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.910289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.910440 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:02.910833 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:02.910840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:02.910845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:02.911226 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:02.911622 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:02.911899 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:02:02.912316 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:02:02.913570 21998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:03.042567 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21875
I20260929 16:02:03.065304 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22007
I20260929 16:02:03.085065 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:03.066240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.5.117.3:43107)
0929 16:02:03.066460 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:03.066465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:03.066492 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:02:03.066650 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:03.066654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:03.066731 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:03.066823 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:03.068366 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:03.069163 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:03.070517 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:03.070904 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:03.071007 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:03.078168 (+ 7161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:03.078189 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:03.078208 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:03.078262 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:03.081478 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:03.082323 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:03.082333 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:03.082336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:03.082570 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:03.083110 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:03.083115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:03.083117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:03.083544 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:03.083808 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:03.084271 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:02:03.084723 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:02:03.085951 22130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:03.187165 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22007
I20260929 16:02:03.207026 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 21653
I20260929 16:02:03.208424 21716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:03.334445 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 21653
I20260929 16:02:23.362198 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--webserver_interface=127.5.117.62
--webserver_port=46335
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:23.749236 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:23.749547 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:23.749665 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:23.756894 22184 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:02:23.756984 22184 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:02:23.757086 22184 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:02:23.757203 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:23.757300 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:23.757449 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:23.757519 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:23.757601 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:23.767230 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=46335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:23.768805 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:23.770349 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:23.784098 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.784171 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.784137 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:23.784809 22184 server_base.cc:1034] running on GCE node
I20260929 16:02:23.785692 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:23.787137 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:23.788399 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790697743788318 us; error 32 us; skew 500 ppm
I20260929 16:02:23.788774 22184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:23 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697743, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:23.798899 22184 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:02:23.800654 22184 webserver.cc:460] Webserver started at http://127.5.117.62:46335/ using document root <none> and password file <none>
I20260929 16:02:23.801476 22184 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:23.801612 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:23.808043 22184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:02:23.811017 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.812489 22184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:23.812677 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "fd82de88e6dd4e6393aa16e939b49bcb"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "8f4901ae47d34c33484de88f6fc55760"
server_key_iv: "d5aca71fbc24bd36935848e00ed19138"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.813799 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:23.827024 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:23.830858 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:23.831094 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:23.851605 22184 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:39343
I20260929 16:02:23.851649 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:39343 every 8 connection(s)
I20260929 16:02:23.853196 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:23.856755 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22184
I20260929 16:02:23.857765 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:36091
--local_ip_for_outbound_sockets=127.5.117.1
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=36427
--webserver_interface=127.5.117.1
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:02:23.860872 22251 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:02:23.864389 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap starting.
I20260929 16:02:23.891964 22251 log.cc:824] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:23.911736 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:02:23.912669 22251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Bootstrap complete.
I20260929 16:02:23.925175 22251 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:23.926410 22251 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Initialized, Role: FOLLOWER
I20260929 16:02:23.927397 22251 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:23.927711 22251 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:23.927860 22251 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:23.928125 22251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:02:23.933246 22251 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:23.934001 22251 leader_election.cc:302] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd82de88e6dd4e6393aa16e939b49bcb; no voters:
I20260929 16:02:23.934918 22251 leader_election.cc:288] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:02:23.935115 22255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:02:23.936445 22255 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [term 3 LEADER]: Becoming Leader. State: Replica: fd82de88e6dd4e6393aa16e939b49bcb, State: Running, Role: LEADER
I20260929 16:02:23.937278 22255 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } }
I20260929 16:02:23.937873 22251 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:23.940312 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd82de88e6dd4e6393aa16e939b49bcb. Latest consensus state: current_term: 3 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:02:23.940775 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:23.940496 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd82de88e6dd4e6393aa16e939b49bcb" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 39343 } } }
I20260929 16:02:23.940948 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:23.953245 22262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:23.963424 22262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:23.965152 22262 catalog_manager.cc:1259] Loaded cluster ID: 56f52f1a96c54476bb7c2957200260f5
I20260929 16:02:23.965317 22262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:23.970126 22262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:23.972013 22262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Loaded TSK: 0
I20260929 16:02:23.972281 22262 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd82de88e6dd4e6393aa16e939b49bcb: Loaded TSK: 1
F20260929 16:02:23.973541 22262 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faf780e09ad google::LogMessage::Fail() at ??:0
@ 0x7faf780e388a google::LogMessage::SendToLog() at ??:0
@ 0x7faf780e04de google::LogMessage::Flush() at ??:0
@ 0x7faf780e1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faf7be447d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faf7be3373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faf8db0f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faf8db0e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faf8db9a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7faf8db9a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7faf8dbe45e9 std::function<>::operator()() at ??:0
@ 0x7faf8db12cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7faf8db10c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faf8db98610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7faf8db98439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faf7ad2c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faf7ad2f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7faf7ad2f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faf7ad0aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7faf7dfe46db start_thread at ??:0
@ 0x7faf76f7a61f clone at ??:0
W20260929 16:02:24.311247 22253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:24.311729 22253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:24.311920 22253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:24.321154 22253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:24.321365 22253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:24.321578 22253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:24.333928 22253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:36091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=36427
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:24.335842 22253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:24.337512 22253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:24.353168 22298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:24.353231 22299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:24.353106 22301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:24.355155 22253 server_base.cc:1034] running on GCE node
I20260929 16:02:24.355870 22253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:24.356995 22253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:24.358256 22253 hybrid_clock.cc:648] HybridClock initialized: now 1790697744358223 us; error 49 us; skew 500 ppm
I20260929 16:02:24.358633 22253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:24 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:24.362916 22253 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:02:24.364667 22253 webserver.cc:460] Webserver started at http://127.5.117.1:36427/ using document root <none> and password file <none>
I20260929 16:02:24.365465 22253 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:24.365628 22253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:24.371888 22253 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:02:24.375021 22308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:24.376477 22253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:24.376689 22253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "f529766f1a2b44e4b7a0b1191361d1bd"
format_stamp: "Formatted at 2026-09-29 16:01:37 on dist-test-slave-b69v"
server_key: "940d9d1e4884205c6ff808c3cfdfc1c6"
server_key_iv: "1524151bc912abcf781b48acefd1f5bf"
server_key_version: "encryptionkey@0"
I20260929 16:02:24.377795 22253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:24.439571 22253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:24.444375 22253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:24.444692 22253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:24.445871 22253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:24.447520 22253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:24.447656 22253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:24.447815 22253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:24.447917 22253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:24.491616 22253 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36091
I20260929 16:02:24.491668 22420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36091 every 8 connection(s)
I20260929 16:02:24.493403 22253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:24.494478 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22253
I20260929 16:02:24.495468 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:38741
--local_ip_for_outbound_sockets=127.5.117.2
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=36467
--webserver_interface=127.5.117.2
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:02:24.500763 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.497873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:49997)
0929 16:02:24.498953 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.499272 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":390,"threads_started":1}
W20260929 16:02:24.501701 22421 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39343 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:39343: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
I20260929 16:02:24.502875 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.502458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:58527)
0929 16:02:24.502637 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.502723 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:02:24.504019 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.503632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:54157)
0929 16:02:24.503775 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.503882 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 16:02:24.504558 22421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:02:24.875172 22424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:24.875470 22424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:24.875581 22424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:24.882985 22424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:24.883093 22424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:24.883209 22424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:24.893034 22424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:38741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=36467
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:24.894476 22424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:24.895964 22424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:24.910269 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:24.910310 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:24.910351 22432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:24.911538 22424 server_base.cc:1034] running on GCE node
I20260929 16:02:24.912238 22424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:24.913363 22424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:24.914593 22424 hybrid_clock.cc:648] HybridClock initialized: now 1790697744914545 us; error 45 us; skew 500 ppm
I20260929 16:02:24.914985 22424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:24 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697744, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:24.918403 22424 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:02:24.919991 22424 webserver.cc:460] Webserver started at http://127.5.117.2:36467/ using document root <none> and password file <none>
I20260929 16:02:24.920812 22424 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:24.920961 22424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:24.926704 22424 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 16:02:24.929529 22439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:24.930872 22424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:24.931039 22424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "40d1492dc7174749a875ab3283d7e9a2"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "929461235ef9ab83b4d117aa59f1621f"
server_key_iv: "949e108c23423039f97cb420fcd038e4"
server_key_version: "encryptionkey@0"
I20260929 16:02:24.932085 22424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:24.956566 22424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:24.961128 22424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:24.961432 22424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:24.962587 22424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:24.964249 22424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:24.964385 22424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:24.964516 22424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:24.964627 22424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:25.001854 22424 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:38741
I20260929 16:02:25.001916 22551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:38741 every 8 connection(s)
I20260929 16:02:25.003636 22424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:25.009305 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22424
I20260929 16:02:25.009785 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.007182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:38825)
0929 16:02:25.007992 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.008382 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":318,"threads_started":1}
I20260929 16:02:25.010270 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:43107
--local_ip_for_outbound_sockets=127.5.117.3
--tserver_master_addrs=127.5.117.62:39343
--webserver_port=35365
--webserver_interface=127.5.117.3
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:25.010663 22552 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39343 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:39343: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
I20260929 16:02:25.011757 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.011263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:34575)
0929 16:02:25.011475 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.011578 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:02:25.012814 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.012369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:33791)
0929 16:02:25.012544 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.012651 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:02:25.013083 22552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:02:25.394663 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:25.394987 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:25.395092 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:25.401997 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:25.402107 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:25.402223 22555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:25.412453 22555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:43107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=35365
--tserver_master_addrs=127.5.117.62:39343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:25.414266 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:25.415799 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:25.430944 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:25.431111 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:25.431143 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:25.432602 22555 server_base.cc:1034] running on GCE node
I20260929 16:02:25.433247 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:25.434170 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:25.435392 22555 hybrid_clock.cc:648] HybridClock initialized: now 1790697745435359 us; error 33 us; skew 500 ppm
I20260929 16:02:25.435757 22555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:25 dist-test-slave-b69v krb5kdc[21101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697745, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:25.439256 22555 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:02:25.440982 22555 webserver.cc:460] Webserver started at http://127.5.117.3:35365/ using document root <none> and password file <none>
I20260929 16:02:25.441793 22555 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:25.441942 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:25.448145 22555 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:02:25.451203 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:25.453207 22555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:02:25.453361 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "4277236c8caa4af78fd90f83a7a85d0c"
format_stamp: "Formatted at 2026-09-29 16:01:38 on dist-test-slave-b69v"
server_key: "07c5fdaeb4659df667569c5e083f8ab8"
server_key_iv: "8d00194c9a2b78aa6b108462e945f5fe"
server_key_version: "encryptionkey@0"
I20260929 16:02:25.454521 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:25.494534 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:25.500190 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:25.500572 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:25.501768 22555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:25.503316 22555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:25.503427 22555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:25.503588 22555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:25.503696 22555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:25.506287 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.505304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:44155)
0929 16:02:25.505924 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.506026 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":309,"threads_started":1}
I20260929 16:02:25.540975 22555 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43107
I20260929 16:02:25.541061 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43107 every 8 connection(s)
I20260929 16:02:25.542582 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:25.546581 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22555
I20260929 16:02:25.547813 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.545816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:53945)
0929 16:02:25.546649 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.546944 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":371,"threads_started":1}
W20260929 16:02:25.548630 22684 heartbeater.cc:644] Failed to heartbeat to 127.5.117.62:39343 (0 consecutive failures): Network error: Failed to ping master at 127.5.117.62:39343: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
I20260929 16:02:25.549378 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.549073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:36451)
0929 16:02:25.549213 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.549275 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 16:02:25.550170 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:25.549890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:37595)
0929 16:02:25.549997 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:25.550067 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 16:02:25.550401 22684 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:02:26.014487 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:26.013635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:38907)
0929 16:02:26.014112 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:26.014205 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":227,"threads_started":1}
I20260929 16:02:26.508184 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:26.507168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:41697)
0929 16:02:26.507792 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:26.507978 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":406,"threads_started":1}
I20260929 16:02:26.551802 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:26.550944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:39677)
0929 16:02:26.551448 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:26.551558 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
I20260929 16:02:27.016247 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.015472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:33491)
0929 16:02:27.015914 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:27.016001 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":216,"threads_started":1}
I20260929 16:02:27.510365 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.509387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:47609)
0929 16:02:27.510003 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:27.510111 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":320,"threads_started":1}
I20260929 16:02:27.554211 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.553036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:49419)
0929 16:02:27.553770 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:27.553910 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":361,"threads_started":1}
I20260929 16:02:28.018004 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.017187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:38957)
0929 16:02:28.017684 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.017775 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
I20260929 16:02:28.512651 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.511586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:48913)
0929 16:02:28.512201 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.512360 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":284,"threads_started":1}
I20260929 16:02:28.556838 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.555631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:49813)
0929 16:02:28.556496 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.556621 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":453,"threads_started":1}
I20260929 16:02:29.020031 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:29.019173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:60209)
0929 16:02:29.019686 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:29.019781 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
I20260929 16:02:29.514758 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:29.513873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:34803)
0929 16:02:29.514416 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:29.514544 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260929 16:02:29.558743 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:29.557893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:35645)
0929 16:02:29.558423 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:29.558530 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260929 16:02:30.022324 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:30.021291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:40127)
0929 16:02:30.022017 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:30.022111 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":441,"threads_started":1}
I20260929 16:02:30.516835 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:30.515900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:44877)
0929 16:02:30.516512 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:30.516624 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":356,"threads_started":1}
I20260929 16:02:30.560997 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:30.560157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:50835)
0929 16:02:30.560692 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:30.560784 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":295,"threads_started":1}
I20260929 16:02:31.024549 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.023568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:36343)
0929 16:02:31.024212 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.024325 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":317,"threads_started":1}
I20260929 16:02:31.519091 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.517989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:52867)
0929 16:02:31.518596 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.518701 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":272,"threads_started":1}
I20260929 16:02:31.563150 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.562196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:58661)
0929 16:02:31.562759 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.562867 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":305,"threads_started":1}
I20260929 16:02:32.026571 22706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.025610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:54101)
0929 16:02:32.026197 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.026291 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":255,"threads_started":1}
I20260929 16:02:32.521179 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.520341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:55159)
0929 16:02:32.520875 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.520974 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":284,"threads_started":1}
I20260929 16:02:32.565376 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.564383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:45305)
0929 16:02:32.565011 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.565113 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":281,"threads_started":1}
I20260929 16:02:33.028443 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:33.027621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:40467)
0929 16:02:33.028133 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:33.028225 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":233,"threads_started":1}
I20260929 16:02:33.523188 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:33.522306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:51863)
0929 16:02:33.522864 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:33.522951 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":305,"threads_started":1}
I20260929 16:02:33.567154 22711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:33.566398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:33945)
0929 16:02:33.566869 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:33.566966 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":269,"threads_started":1}
I20260929 16:02:34.030109 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:34.029351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:51069)
0929 16:02:34.029833 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:34.029916 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260929 16:02:34.525609 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:34.524666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:46735)
0929 16:02:34.525213 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:34.525320 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":304,"threads_started":1}
I20260929 16:02:34.569546 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:34.568232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:59067)
0929 16:02:34.568975 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:34.569292 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":364,"threads_started":1}
I20260929 16:02:35.031840 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.031046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:41873)
0929 16:02:35.031539 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:35.031637 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":284,"threads_started":1}
I20260929 16:02:35.527756 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.526816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:60103)
0929 16:02:35.527401 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:35.527505 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":300,"threads_started":1}
I20260929 16:02:35.571664 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.570817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:48529)
0929 16:02:35.571353 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:35.571450 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":313,"threads_started":1}
I20260929 16:02:36.033582 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.032787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:34657)
0929 16:02:36.033243 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.033360 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260929 16:02:36.529614 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.528891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:49743)
0929 16:02:36.529359 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.529444 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260929 16:02:36.573643 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.572827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:35665)
0929 16:02:36.573301 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.573395 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":251,"threads_started":1}
I20260929 16:02:37.035252 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:37.034502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:36937)
0929 16:02:37.034937 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:37.035026 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260929 16:02:37.532419 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:37.531100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:37047)
0929 16:02:37.531965 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:37.532167 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":339,"threads_started":1}
I20260929 16:02:37.575673 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:37.574814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:53943)
0929 16:02:37.575359 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:37.575463 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
I20260929 16:02:38.037245 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.036308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:57165)
0929 16:02:38.036887 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:38.036995 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":323,"threads_started":1}
I20260929 16:02:38.534482 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.533587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:54327)
0929 16:02:38.534157 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:38.534248 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":324,"threads_started":1}
I20260929 16:02:38.577809 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.576867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:40429)
0929 16:02:38.577482 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:38.577586 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":369,"threads_started":1}
I20260929 16:02:39.038841 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.038099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:56419)
0929 16:02:39.038555 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.038638 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260929 16:02:39.537058 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.535982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:56523)
0929 16:02:39.536685 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.536792 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":461,"threads_started":1}
I20260929 16:02:39.579821 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.578942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:51123)
0929 16:02:39.579490 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.579590 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260929 16:02:40.040904 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.040006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.2:36243)
0929 16:02:40.040561 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:40.040692 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
I20260929 16:02:40.538996 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.538129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.1:58131)
0929 16:02:40.538677 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:40.538779 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":293,"threads_started":1}
I20260929 16:02:40.555275 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22253
I20260929 16:02:40.577282 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.556821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.5.117.1:36091)
0929 16:02:40.557594 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:40.557613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:40.557668 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:02:40.559999 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:40.560010 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:40.560598 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:40.561006 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:40.562672 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.563984 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.565339 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.566040 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.566241 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:40.569254 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:40.569262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:40.569277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:40.569326 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:40.573266 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.574230 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.574238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.574246 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.574435 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.574841 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.574847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.574850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.575487 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:40.575870 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:40.576482 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 16:02:40.576953 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":516,"threads_started":1}
I20260929 16:02:40.578926 22416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:40.582376 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.581277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:39343 (local address 127.5.117.3:41399)
0929 16:02:40.581888 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:40.582061 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.62:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
I20260929 16:02:40.673368 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22253
I20260929 16:02:40.697711 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22424
I20260929 16:02:40.718521 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.698943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.5.117.2:38741)
0929 16:02:40.699598 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:40.699617 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:40.699665 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:02:40.701593 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:40.701606 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:40.702141 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:40.702500 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:40.704233 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.705727 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.707508 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.708493 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.708657 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:40.711639 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:40.711647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:40.711666 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:40.711708 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:40.714975 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.715747 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.715753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.715760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.715937 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.716404 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.716408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.716410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.717009 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:40.717273 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:40.717704 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:02:40.718190 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":297,"threads_started":1}
I20260929 16:02:40.720203 22547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:40.813329 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22424
I20260929 16:02:40.835285 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22555
I20260929 16:02:40.855060 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.836463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.5.117.3:43107)
0929 16:02:40.837044 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:40.837064 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:40.837133 (+ 69us) server_negotiation.cc:407] Connection header received
0929 16:02:40.839022 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:40.839033 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:40.839563 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:40.839970 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:40.841708 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.842972 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.844380 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:40.845056 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:40.845288 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:40.848328 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:40.848336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:40.848354 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:40.848394 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:40.851242 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.851990 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.851997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.852004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.852291 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:40.852740 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:40.852745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:40.852748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:40.853366 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:40.853740 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:40.854208 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 16:02:40.854708 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
I20260929 16:02:40.857189 22679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:40.948593 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22555
2026-09-29T16:02:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65730 ms)
[----------] 30 tests from SecurityITest (319189 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:41 dist-test-slave-b69v krb5kdc[22741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:41 dist-test-slave-b69v krb5kdc[22741](info): set up 2 sockets
Sep 29 16:02:41 dist-test-slave-b69v krb5kdc[22741](info): commencing operation
krb5kdc: starting...
W20260929 16:02:42.065397 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:02:42 dist-test-slave-b69v krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:02:42.356101 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40177
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:42.724740 22754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:42.725035 22754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:42.725209 22754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:42.732131 22754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:42.732259 22754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:42.732366 22754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:42.732474 22754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:42.732564 22754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:42.742096 22754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:40177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:42.743495 22754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:42.744910 22754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:42.758039 22759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:42.758073 22760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:42.758121 22762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:42.758385 22754 server_base.cc:1034] running on GCE node
I20260929 16:02:42.759328 22754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:42.760743 22754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:42.761974 22754 hybrid_clock.cc:648] HybridClock initialized: now 1790697762761930 us; error 47 us; skew 500 ppm
I20260929 16:02:42.762323 22754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:42 dist-test-slave-b69v krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:42.765409 22754 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:02:42.766969 22754 webserver.cc:460] Webserver started at http://127.5.117.62:43087/ using document root <none> and password file <none>
I20260929 16:02:42.767648 22754 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:42.767792 22754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:42.768169 22754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:42.771109 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "e92816e7a23d4f6ea719db230f34b553"
format_stamp: "Formatted at 2026-09-29 16:02:42 on dist-test-slave-b69v"
server_key: "e5f086c0ac6199bcfe01124765f8f266"
server_key_iv: "bcee44fc75420398b0a2fdb6d1105e1a"
server_key_version: "encryptionkey@0"
I20260929 16:02:42.771925 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "e92816e7a23d4f6ea719db230f34b553"
format_stamp: "Formatted at 2026-09-29 16:02:42 on dist-test-slave-b69v"
server_key: "e5f086c0ac6199bcfe01124765f8f266"
server_key_iv: "bcee44fc75420398b0a2fdb6d1105e1a"
server_key_version: "encryptionkey@0"
I20260929 16:02:42.777374 22754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:02:42.780875 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:42.782173 22754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:42.782351 22754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "e92816e7a23d4f6ea719db230f34b553"
format_stamp: "Formatted at 2026-09-29 16:02:42 on dist-test-slave-b69v"
server_key: "e5f086c0ac6199bcfe01124765f8f266"
server_key_iv: "bcee44fc75420398b0a2fdb6d1105e1a"
server_key_version: "encryptionkey@0"
I20260929 16:02:42.782549 22754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:42.797880 22754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:42.801831 22754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:42.802157 22754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:42.825542 22754 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:40177
I20260929 16:02:42.825567 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:40177 every 8 connection(s)
I20260929 16:02:42.827261 22754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:42.832394 22821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:42.837069 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22754
I20260929 16:02:42.837307 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:02:42.838079 5588 external_mini_cluster.cc:1428] Setting key cfdaacea864bb396d42b386d4fd2d84c
I20260929 16:02:42.846324 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: Bootstrap starting.
I20260929 16:02:42.850477 22821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:42.851922 22821 log.cc:824] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:42.854984 22821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: No bootstrap required, opened a new log
Sep 29 16:02:42 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:42.865962 22821 raft_consensus.cc:348] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } }
I20260929 16:02:42.866407 22821 raft_consensus.cc:374] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:42.866530 22821 raft_consensus.cc:729] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92816e7a23d4f6ea719db230f34b553, State: Initialized, Role: FOLLOWER
I20260929 16:02:42.866708 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:42.840479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.5.117.62:40177)
0929 16:02:42.841367 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:42.841382 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:42.841455 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:02:42.844315 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:42.844345 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:42.844855 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:42.845296 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:42.847210 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.849420 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.850913 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.852001 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.852266 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:42.857019 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:42.857032 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:42.857053 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:42.857097 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:42.861649 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.862755 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.862765 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.862774 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.862986 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.863444 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.863449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.863452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.864178 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:42.864518 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:42.864879 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:02:42.865308 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":375,"threads_started":1}
I20260929 16:02:42.867208 22821 consensus_queue.cc:260] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } }
I20260929 16:02:42.867491 22821 raft_consensus.cc:388] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:42.867654 22821 raft_consensus.cc:482] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:42.867841 22821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:42.870236 22821 raft_consensus.cc:504] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } }
I20260929 16:02:42.870740 22821 leader_election.cc:302] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92816e7a23d4f6ea719db230f34b553; no voters:
I20260929 16:02:42.871337 22821 leader_election.cc:288] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:42.871475 22826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:42.872550 22826 raft_consensus.cc:686] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [term 1 LEADER]: Becoming Leader. State: Replica: e92816e7a23d4f6ea719db230f34b553, State: Running, Role: LEADER
I20260929 16:02:42.873224 22826 consensus_queue.cc:237] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } }
I20260929 16:02:42.874714 22821 sys_catalog.cc:564] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:42.876221 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e92816e7a23d4f6ea719db230f34b553. Latest consensus state: current_term: 1 leader_uuid: "e92816e7a23d4f6ea719db230f34b553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } } }
I20260929 16:02:42.876896 22827 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e92816e7a23d4f6ea719db230f34b553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92816e7a23d4f6ea719db230f34b553" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 40177 } } }
I20260929 16:02:42.877185 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:42.877269 22827 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:42.879427 22831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:42.886170 22831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:42.897629 22831 catalog_manager.cc:1347] Generated new cluster ID: 4536c49999824858a02cac67ccb21802
I20260929 16:02:42.897758 22831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:42.921876 22831 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:42.924023 22831 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:42.933820 22831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: Generated new TSK 0
I20260929 16:02:42.934934 22831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:02:43.015318 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:43.404210 22847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:43.404508 22847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:43.404625 22847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:43.411881 22847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:43.411978 22847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:43.412164 22847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:43.421716 22847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:43.422979 22847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:43.424520 22847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:43.439563 22852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:43.439895 22853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:43.439944 22855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:43.446105 22847 server_base.cc:1034] running on GCE node
I20260929 16:02:43.446846 22847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:43.447937 22847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:43.449157 22847 hybrid_clock.cc:648] HybridClock initialized: now 1790697763449112 us; error 33 us; skew 500 ppm
I20260929 16:02:43.449527 22847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:43 dist-test-slave-b69v krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:43.458777 22847 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:02:43.460669 22847 webserver.cc:460] Webserver started at http://127.5.117.1:38891/ using document root <none> and password file <none>
I20260929 16:02:43.461545 22847 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:43.461690 22847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:43.462065 22847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:43.465575 22847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1db2fcd9e1cd4483860e4dae3f3433aa"
format_stamp: "Formatted at 2026-09-29 16:02:43 on dist-test-slave-b69v"
server_key: "4d317c60655d0a1104c9a3bbcb919e7c"
server_key_iv: "3a4f249dd33ef86a6837a0c9ccb064e6"
server_key_version: "encryptionkey@0"
I20260929 16:02:43.466485 22847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1db2fcd9e1cd4483860e4dae3f3433aa"
format_stamp: "Formatted at 2026-09-29 16:02:43 on dist-test-slave-b69v"
server_key: "4d317c60655d0a1104c9a3bbcb919e7c"
server_key_iv: "3a4f249dd33ef86a6837a0c9ccb064e6"
server_key_version: "encryptionkey@0"
I20260929 16:02:43.472462 22847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:02:43.476433 22862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:43.477892 22847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:43.478085 22847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1db2fcd9e1cd4483860e4dae3f3433aa"
format_stamp: "Formatted at 2026-09-29 16:02:43 on dist-test-slave-b69v"
server_key: "4d317c60655d0a1104c9a3bbcb919e7c"
server_key_iv: "3a4f249dd33ef86a6837a0c9ccb064e6"
server_key_version: "encryptionkey@0"
I20260929 16:02:43.478300 22847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:43.490379 22847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:43.495227 22847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:43.495594 22847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:43.496819 22847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:43.498371 22847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:43.498481 22847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:43.498641 22847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:43.498714 22847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:43.537132 22847 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:42873
I20260929 16:02:43.537184 22974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:42873 every 8 connection(s)
I20260929 16:02:43.538902 22847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:43.549191 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22847
I20260929 16:02:43.549479 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:43.550233 5588 external_mini_cluster.cc:1428] Setting key 671b564a4f77203b2ee38991e1bbb456
Sep 29 16:02:43 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697763, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:43.564627 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:43.542656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:53807 (local address 127.5.117.62:40177)
0929 16:02:43.543254 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:43.543259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:43.544567 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 16:02:43.546901 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:43.546908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:43.547034 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:43.547175 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:43.549227 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:43.550214 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:43.552304 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:43.553040 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:43.553212 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:43.557695 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:43.557702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:43.557705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:43.557746 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:43.560814 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:43.561801 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:43.561806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:43.561808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:43.561947 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:43.562445 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:43.562452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:43.562459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:43.562770 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:43.562958 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:43.563846 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 16:02:43.564174 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":316,"threads_started":1}
I20260929 16:02:43.565884 22978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:43.543218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40177 (local address 127.5.117.1:53807)
0929 16:02:43.544308 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:43.544379 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:43.546515 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:43.547497 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:43.547508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:43.548265 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:43.549009 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:43.549030 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:43.550413 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:43.550419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:43.552128 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:43.552151 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:43.553338 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:43.553347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:43.553651 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:43.554552 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:43.554580 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:43.557351 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:43.561031 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:43.561044 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:43.561077 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:43.561646 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:43.562073 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:43.562079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:43.562082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:43.562310 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:43.562921 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:43.562931 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:43.563387 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 16:02:43.563707 (+ 320us) client_negotiation.cc:240] Negotiation successful
0929 16:02:43.564468 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":396,"threads_started":1}
I20260929 16:02:43.567972 22975 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40177
I20260929 16:02:43.568419 22975 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:43.569528 22975 heartbeater.cc:511] Master 127.5.117.62:40177 requested a full tablet report, sending...
I20260929 16:02:43.572702 22786 ts_manager.cc:194] Registered new tserver with Master: 1db2fcd9e1cd4483860e4dae3f3433aa (127.5.117.1:42873)
I20260929 16:02:43.574496 22786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:53807
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:02:43.628501 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:44.002842 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:44.003149 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:44.003331 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:44.010380 22981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:44.010469 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:44.010633 22981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:44.019825 22981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:44.021183 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:44.022610 22981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:44.036437 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.036533 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.036507 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:44.037654 22981 server_base.cc:1034] running on GCE node
I20260929 16:02:44.038271 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:44.039170 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:44.040418 22981 hybrid_clock.cc:648] HybridClock initialized: now 1790697764040381 us; error 49 us; skew 500 ppm
I20260929 16:02:44.040771 22981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:44 dist-test-slave-b69v krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:44.044220 22981 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:02:44.045601 22981 webserver.cc:460] Webserver started at http://127.5.117.2:35655/ using document root <none> and password file <none>
I20260929 16:02:44.046279 22981 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:44.046381 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:44.046674 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:44.049582 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "b80f60d5141741febc932147796e3cf0"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "53941134f60491dafeecada32f96e3da"
server_key_iv: "fbe99d3f9265e12ea4abbfda3fc74f0b"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.050415 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "b80f60d5141741febc932147796e3cf0"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "53941134f60491dafeecada32f96e3da"
server_key_iv: "fbe99d3f9265e12ea4abbfda3fc74f0b"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.057166 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:02:44.064661 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.066352 22981 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260929 16:02:44.066534 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "b80f60d5141741febc932147796e3cf0"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "53941134f60491dafeecada32f96e3da"
server_key_iv: "fbe99d3f9265e12ea4abbfda3fc74f0b"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.066761 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:44.082765 22981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:44.087195 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:44.087525 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:44.088567 22981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:44.089869 22981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:44.089947 22981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.090030 22981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:44.090077 22981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.130240 22981 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:41409
I20260929 16:02:44.130283 23108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:41409 every 8 connection(s)
I20260929 16:02:44.131752 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:44.136255 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 22981
I20260929 16:02:44.136487 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:44.137185 5588 external_mini_cluster.cc:1428] Setting key 79be3b1edc2ebbf0d4c6878905bcc9f0
Sep 29 16:02:44 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:44.156468 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.135003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:43431 (local address 127.5.117.62:40177)
0929 16:02:44.135541 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.135550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.137506 (+ 1956us) server_negotiation.cc:407] Connection header received
0929 16:02:44.139232 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.139241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.139331 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.139490 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.141718 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.142527 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.145071 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.145533 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.145637 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.150370 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.150375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.150379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:44.150416 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.153242 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.154007 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.154015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.154018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.154177 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.154620 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.154630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.154632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.154898 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.155083 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.155763 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.156002 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
I20260929 16:02:44.157423 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.135402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40177 (local address 127.5.117.2:43431)
0929 16:02:44.136402 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.136471 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.138950 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.139750 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.139764 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.140577 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:44.141435 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.141532 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.142698 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.142707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.144894 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.144912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.145765 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.145773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.146214 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.147158 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:44.147192 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:44.150127 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:44.153382 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:44.153390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:44.153404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:44.153869 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:44.154301 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:44.154306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:44.154309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:44.154497 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:44.155036 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:44.155043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:44.155387 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.155658 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.156191 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":370,"threads_started":1}
I20260929 16:02:44.158963 23109 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40177
I20260929 16:02:44.159319 23109 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:44.160306 23109 heartbeater.cc:511] Master 127.5.117.62:40177 requested a full tablet report, sending...
I20260929 16:02:44.162072 22786 ts_manager.cc:194] Registered new tserver with Master: b80f60d5141741febc932147796e3cf0 (127.5.117.2:41409)
I20260929 16:02:44.162981 22786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:43431
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:02:44.203346 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:02:44.579344 22975 heartbeater.cc:503] Master 127.5.117.62:40177 was elected leader, sending a full tablet report...
W20260929 16:02:44.587502 23115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:44.587797 23115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:44.587961 23115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:44.594870 23115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:44.595032 23115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:44.595209 23115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:44.604395 23115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:40177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:44.605751 23115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:44.607208 23115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:44.620205 23120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.620241 23121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.620401 23123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:44.621111 23115 server_base.cc:1034] running on GCE node
I20260929 16:02:44.621640 23115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:44.622493 23115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:44.623677 23115 hybrid_clock.cc:648] HybridClock initialized: now 1790697764623639 us; error 30 us; skew 500 ppm
I20260929 16:02:44.623961 23115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:44 dist-test-slave-b69v krb5kdc[22741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:44.627070 23115 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:02:44.628486 23115 webserver.cc:460] Webserver started at http://127.5.117.3:33467/ using document root <none> and password file <none>
I20260929 16:02:44.629148 23115 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:44.629284 23115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:44.629587 23115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:44.632539 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "9d553c6107c4446b82ef6957c3d0136f"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "b7afa1f34cc017454de672268ff53379"
server_key_iv: "ff0c602fe4b360d37c704c5b37eb6807"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.633315 23115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "9d553c6107c4446b82ef6957c3d0136f"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "b7afa1f34cc017454de672268ff53379"
server_key_iv: "ff0c602fe4b360d37c704c5b37eb6807"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.638128 23115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:02:44.641461 23130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.642599 23115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:02:44.642756 23115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "9d553c6107c4446b82ef6957c3d0136f"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-b69v"
server_key: "b7afa1f34cc017454de672268ff53379"
server_key_iv: "ff0c602fe4b360d37c704c5b37eb6807"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.642957 23115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:44.661473 23115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:44.665323 23115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:44.665620 23115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:44.666613 23115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:44.667950 23115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:44.668051 23115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.668224 23115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:44.668293 23115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.704681 23115 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:36835
I20260929 16:02:44.704715 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:36835 every 8 connection(s)
I20260929 16:02:44.706198 23115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:44.707818 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23115
I20260929 16:02:44.708027 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:44.708657 5588 external_mini_cluster.cc:1428] Setting key 9d858bd966ea3d6f67cc580ca5df1953
Sep 29 16:02:44 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:44.727604 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.709187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:54933 (local address 127.5.117.62:40177)
0929 16:02:44.709951 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.709959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.710745 (+ 786us) server_negotiation.cc:407] Connection header received
0929 16:02:44.713275 (+ 2530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.713280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.713377 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.713530 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.715073 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.715928 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.717491 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.717874 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.717994 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.721504 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.721510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.721513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:44.721554 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.724787 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.725534 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.725538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.725540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.725681 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.726084 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.726087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.726089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.726366 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.726488 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.727155 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.727321 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":428,"threads_started":1}
I20260929 16:02:44.728714 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.709697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:40177 (local address 127.5.117.3:54933)
0929 16:02:44.710594 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.710632 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.713007 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.713783 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.713791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.714305 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:44.714906 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.714923 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.716102 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.716107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.717364 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.717374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.718088 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.718096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.718319 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.718983 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:44.719008 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:44.721276 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:44.724908 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:44.724914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:44.724927 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:44.725426 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:44.725794 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:44.725797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:44.725800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:44.725989 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:44.726469 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:44.726475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:44.726812 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.727073 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.727474 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":347,"threads_started":1}
I20260929 16:02:44.730094 23243 heartbeater.cc:347] Connected to a master server at 127.5.117.62:40177
I20260929 16:02:44.730403 23243 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:44.731171 23243 heartbeater.cc:511] Master 127.5.117.62:40177 requested a full tablet report, sending...
I20260929 16:02:44.732664 22786 ts_manager.cc:194] Registered new tserver with Master: 9d553c6107c4446b82ef6957c3d0136f (127.5.117.3:36835)
I20260929 16:02:44.733518 22786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:54933
I20260929 16:02:44.737249 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:44.754741 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.742097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.5.117.62:40177)
0929 16:02:44.742320 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.742328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.742756 (+ 428us) server_negotiation.cc:407] Connection header received
0929 16:02:44.742918 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.742922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.742988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.743067 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.744686 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.745446 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.746943 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.747348 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.747434 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.749841 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.749845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.749848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:44.749877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.752300 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.752936 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.752940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.752943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.753063 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.753431 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.753434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.753436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.753767 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.753969 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.754229 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.754396 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:02:44.758978 22786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:44.766258 22786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:44.797158 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.779999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:41409 (local address 127.0.0.1:51768)
0929 16:02:44.784472 (+ 4473us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.784486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.784626 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.789535 (+ 4909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.789539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.789563 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.790004 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.790018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.792107 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.792112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.794268 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.794280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.795982 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.795990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.796228 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.796293 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.796405 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.796598 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4355,"mutex_wait_us":118,"thread_start_us":663,"threads_started":1}
I20260929 16:02:44.797791 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.779602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:42873 (local address 127.0.0.1:48690)
0929 16:02:44.781041 (+ 1439us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.781090 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.781254 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.790733 (+ 9479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.790737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.790761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.791135 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.791145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.792782 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.792787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.795113 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.795122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.796937 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.796950 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.797148 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.797180 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.797260 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.797392 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":20,"thread_start_us":4613,"threads_started":1}
I20260929 16:02:44.798152 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.780043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.5.117.1:42873)
0929 16:02:44.790314 (+ 10271us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.790320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.790340 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:02:44.790421 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.790426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.790557 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.790745 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.791304 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.792632 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.795253 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.796679 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.797164 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.797393 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.797465 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.797636 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10079,"thread_start_us":9921,"threads_started":1}
I20260929 16:02:44.798352 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.780574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.5.117.2:41409)
0929 16:02:44.788795 (+ 8221us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.788807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.788842 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:02:44.788982 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.788993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.789243 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.789557 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.790196 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.791933 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.794436 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.795742 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.797668 (+ 1926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.797797 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.797874 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.798054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8032,"thread_start_us":7924,"threads_started":1}
I20260929 16:02:44.798627 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.784656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:36835 (local address 127.0.0.1:52586)
0929 16:02:44.788398 (+ 3742us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.788411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.788553 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.789117 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.789124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.789151 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.789539 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.789550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.792411 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.792419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.794890 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.794905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.796909 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.796921 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.797979 (+ 1058us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.798019 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.798102 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.798246 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3654,"mutex_wait_us":31,"thread_start_us":3547,"threads_started":1}
I20260929 16:02:44.799809 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.785050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.5.117.3:36835)
0929 16:02:44.785962 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.785969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.788538 (+ 2569us) server_negotiation.cc:407] Connection header received
0929 16:02:44.788708 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.788714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.788913 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.789115 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.789800 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.792260 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.795101 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.796656 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.796791 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.797052 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.798145 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.799432 (+ 1287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":550,"threads_started":1}
I20260929 16:02:44.802410 22910 tablet_service.cc:1467] Processing CreateTablet for tablet 3189b8306686492598915de980f225fc (DEFAULT_TABLE table=test-table [id=608cbe95128a4e08a09638071dae236e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:44.802424 23044 tablet_service.cc:1467] Processing CreateTablet for tablet 3189b8306686492598915de980f225fc (DEFAULT_TABLE table=test-table [id=608cbe95128a4e08a09638071dae236e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:44.803275 23178 tablet_service.cc:1467] Processing CreateTablet for tablet 3189b8306686492598915de980f225fc (DEFAULT_TABLE table=test-table [id=608cbe95128a4e08a09638071dae236e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:44.803934 23044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3189b8306686492598915de980f225fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:44.803980 22910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3189b8306686492598915de980f225fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:44.804666 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3189b8306686492598915de980f225fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:44.818435 23262 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Bootstrap starting.
I20260929 16:02:44.819530 23263 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Bootstrap starting.
I20260929 16:02:44.823042 23262 tablet_bootstrap.cc:654] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:44.823462 23263 tablet_bootstrap.cc:654] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:44.824123 23264 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Bootstrap starting.
I20260929 16:02:44.824837 23263 log.cc:824] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:44.825022 23262 log.cc:824] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:44.827643 23264 tablet_bootstrap.cc:654] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:44.828928 23264 log.cc:824] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:44.834704 23263 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: No bootstrap required, opened a new log
I20260929 16:02:44.834888 23262 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: No bootstrap required, opened a new log
I20260929 16:02:44.835011 23263 ts_tablet_manager.cc:1397] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260929 16:02:44.835188 23262 ts_tablet_manager.cc:1397] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260929 16:02:44.835705 23264 tablet_bootstrap.cc:492] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: No bootstrap required, opened a new log
I20260929 16:02:44.835953 23264 ts_tablet_manager.cc:1397] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260929 16:02:44.846275 23263 raft_consensus.cc:348] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.846495 23262 raft_consensus.cc:348] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.846926 23263 raft_consensus.cc:374] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:44.847014 23262 raft_consensus.cc:374] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:44.847069 23263 raft_consensus.cc:729] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80f60d5141741febc932147796e3cf0, State: Initialized, Role: FOLLOWER
I20260929 16:02:44.847188 23262 raft_consensus.cc:729] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db2fcd9e1cd4483860e4dae3f3433aa, State: Initialized, Role: FOLLOWER
I20260929 16:02:44.847944 23263 consensus_queue.cc:260] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.847954 23262 consensus_queue.cc:260] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.849225 23264 raft_consensus.cc:348] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.849759 23264 raft_consensus.cc:374] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:44.849880 23264 raft_consensus.cc:729] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d553c6107c4446b82ef6957c3d0136f, State: Initialized, Role: FOLLOWER
I20260929 16:02:44.850567 23264 consensus_queue.cc:260] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.851955 23243 heartbeater.cc:503] Master 127.5.117.62:40177 was elected leader, sending a full tablet report...
I20260929 16:02:44.852911 23264 ts_tablet_manager.cc:1428] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 16:02:44.853022 23263 ts_tablet_manager.cc:1428] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260929 16:02:44.853530 23109 heartbeater.cc:503] Master 127.5.117.62:40177 was elected leader, sending a full tablet report...
I20260929 16:02:44.854614 23271 raft_consensus.cc:482] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:44.854871 23271 raft_consensus.cc:504] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.855206 23262 ts_tablet_manager.cc:1428] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260929 16:02:44.859745 23271 leader_election.cc:288] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b80f60d5141741febc932147796e3cf0 (127.5.117.2:41409), 9d553c6107c4446b82ef6957c3d0136f (127.5.117.3:36835)
I20260929 16:02:44.862323 23269 raft_consensus.cc:482] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:44.862632 23269 raft_consensus.cc:504] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.864531 23269 leader_election.cc:288] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1db2fcd9e1cd4483860e4dae3f3433aa (127.5.117.1:42873), 9d553c6107c4446b82ef6957c3d0136f (127.5.117.3:36835)
I20260929 16:02:44.866868 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.857415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:52641 (local address 127.5.117.2:41409)
0929 16:02:44.858076 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.858083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.859247 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 16:02:44.859553 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.859559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.859640 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.859714 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.860763 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.861954 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.863955 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.865402 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.865583 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.865707 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.866275 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.866473 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260929 16:02:44.867992 23272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.857150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:41409 (local address 127.5.117.1:52641)
0929 16:02:44.859138 (+ 1988us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.859160 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.859372 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.859967 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.859973 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.860004 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.860537 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.860555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.862105 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.862111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.863784 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.863796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.865862 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.865871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.866062 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.866098 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.867280 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.867499 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1845,"thread_start_us":2848,"threads_started":1}
I20260929 16:02:44.868374 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.858538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:35371 (local address 127.5.117.3:36835)
0929 16:02:44.858750 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.858758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.860914 (+ 2156us) server_negotiation.cc:407] Connection header received
0929 16:02:44.861163 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.861168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.861240 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.861302 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.862097 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.863251 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.865796 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.867167 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.867259 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.867368 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.867952 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.868100 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:02:44.869167 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.858300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:36835 (local address 127.5.117.1:35371)
0929 16:02:44.860815 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.860835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.861007 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.861500 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.861504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.861527 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.861901 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.861912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.863461 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.863468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.865593 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.865607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.867596 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.867602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.867766 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.867809 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.868662 (+ 853us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.868804 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2388,"thread_start_us":2998,"threads_started":1}
I20260929 16:02:44.869345 23064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80f60d5141741febc932147796e3cf0" is_pre_election: true
I20260929 16:02:44.869905 23064 raft_consensus.cc:2454] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db2fcd9e1cd4483860e4dae3f3433aa in term 0.
I20260929 16:02:44.870710 22863 leader_election.cc:302] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db2fcd9e1cd4483860e4dae3f3433aa, b80f60d5141741febc932147796e3cf0; no voters:
I20260929 16:02:44.870630 23198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d553c6107c4446b82ef6957c3d0136f" is_pre_election: true
I20260929 16:02:44.871124 23271 raft_consensus.cc:2781] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:44.871188 23198 raft_consensus.cc:2454] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1db2fcd9e1cd4483860e4dae3f3433aa in term 0.
I20260929 16:02:44.871268 23271 raft_consensus.cc:482] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:44.871368 23271 raft_consensus.cc:3037] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:44.874266 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.864955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:59565 (local address 127.5.117.1:42873)
0929 16:02:44.865133 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.865139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.865768 (+ 629us) server_negotiation.cc:407] Connection header received
0929 16:02:44.866039 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.866043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.866108 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.866171 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.866988 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.868124 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.871450 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.872919 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.873043 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.873150 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.873865 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.874000 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:02:44.874432 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.864879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:42873 (local address 127.5.117.2:59565)
0929 16:02:44.865676 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.865696 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.865873 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.866360 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.866365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.866389 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.866799 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.866810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.868288 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.868293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.871205 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.871223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.873406 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.873412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.873591 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.873712 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.873824 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.873974 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":69,"thread_start_us":408,"threads_started":1}
I20260929 16:02:44.874402 23271 raft_consensus.cc:504] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
I20260929 16:02:44.875233 23271 leader_election.cc:288] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [CANDIDATE]: Term 1 election: Requested vote from peers b80f60d5141741febc932147796e3cf0 (127.5.117.2:41409), 9d553c6107c4446b82ef6957c3d0136f (127.5.117.3:36835)
I20260929 16:02:44.875438 22930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "b80f60d5141741febc932147796e3cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" is_pre_election: true
I20260929 16:02:44.875799 23064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b80f60d5141741febc932147796e3cf0"
I20260929 16:02:44.875946 22930 raft_consensus.cc:2379] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b80f60d5141741febc932147796e3cf0 in current term 1: Already voted for candidate 1db2fcd9e1cd4483860e4dae3f3433aa in this term.
I20260929 16:02:44.875938 23064 raft_consensus.cc:3037] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:44.876331 23198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d553c6107c4446b82ef6957c3d0136f"
I20260929 16:02:44.876575 23198 raft_consensus.cc:3037] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:44.877839 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.866082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:35539 (local address 127.5.117.3:36835)
0929 16:02:44.868612 (+ 2530us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.868617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.870492 (+ 1875us) server_negotiation.cc:407] Connection header received
0929 16:02:44.870734 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.870739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.870824 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.870880 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:44.871780 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.872907 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.875450 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.876802 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.876878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.876975 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.877439 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.877563 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2430,"thread_start_us":3627,"threads_started":1}
I20260929 16:02:44.878403 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.865742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:36835 (local address 127.5.117.2:35539)
0929 16:02:44.870394 (+ 4652us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:44.870419 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:44.870586 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:44.871046 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:44.871050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:44.871074 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:44.871615 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.871629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.873104 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.873108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.875230 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:44.875244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.877163 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.877167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:44.877281 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.877301 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 16:02:44.877404 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:02:44.877534 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4490,"thread_start_us":6318,"threads_started":1}
I20260929 16:02:44.879113 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3189b8306686492598915de980f225fc" candidate_uuid: "b80f60d5141741febc932147796e3cf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d553c6107c4446b82ef6957c3d0136f" is_pre_election: true
I20260929 16:02:44.879987 22998 leader_election.cc:302] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b80f60d5141741febc932147796e3cf0; no voters: 1db2fcd9e1cd4483860e4dae3f3433aa, 9d553c6107c4446b82ef6957c3d0136f
I20260929 16:02:44.880167 23198 raft_consensus.cc:2454] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db2fcd9e1cd4483860e4dae3f3433aa in term 1.
I20260929 16:02:44.880362 23064 raft_consensus.cc:2454] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1db2fcd9e1cd4483860e4dae3f3433aa in term 1.
I20260929 16:02:44.880614 22864 leader_election.cc:302] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1db2fcd9e1cd4483860e4dae3f3433aa, 9d553c6107c4446b82ef6957c3d0136f; no voters:
I20260929 16:02:44.880661 23269 raft_consensus.cc:2726] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:02:44.880972 23271 raft_consensus.cc:2781] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:44.881138 23271 raft_consensus.cc:686] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 LEADER]: Becoming Leader. State: Replica: 1db2fcd9e1cd4483860e4dae3f3433aa, State: Running, Role: LEADER
I20260929 16:02:44.881593 23271 consensus_queue.cc:237] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } }
W20260929 16:02:44.883941 23110 tablet.cc:2369] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:44.886505 22785 catalog_manager.cc:5495] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1db2fcd9e1cd4483860e4dae3f3433aa (127.5.117.1). New cstate: current_term: 1 leader_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1db2fcd9e1cd4483860e4dae3f3433aa" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 42873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 } health_report { overall_health: UNKNOWN } } }
I20260929 16:02:44.914690 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.909570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.5.117.1:42873)
0929 16:02:44.909887 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.909892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.909909 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:02:44.909981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.909984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.910069 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.910185 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:44.910497 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.911211 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.912614 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.913054 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.913145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.913571 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:44.914106 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:44.914214 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.914298 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.914361 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 16:02:44.922554 23198 raft_consensus.cc:1261] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 1 FOLLOWER]: Refusing update from remote peer 1db2fcd9e1cd4483860e4dae3f3433aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:44.922612 23064 raft_consensus.cc:1261] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 1 FOLLOWER]: Refusing update from remote peer 1db2fcd9e1cd4483860e4dae3f3433aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:44.923398 23268 consensus_queue.cc:1035] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d553c6107c4446b82ef6957c3d0136f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:44.923697 23271 consensus_queue.cc:1035] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b80f60d5141741febc932147796e3cf0" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:44.940491 23280 mvcc.cc:205] Tried to move back new op lower bound from 7334698045113929728 to 7334698044958679040. Current Snapshot: MvccSnapshot[applied={T|T < 7334698045113929728}]
I20260929 16:02:44.953039 22786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:44.953405 22786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:44.957279 22786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e92816e7a23d4f6ea719db230f34b553: Sending DeleteTablet for 3 replicas of tablet 3189b8306686492598915de980f225fc
I20260929 16:02:44.958447 22910 tablet_service.cc:1514] Processing DeleteTablet for tablet 3189b8306686492598915de980f225fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:48690
I20260929 16:02:44.958487 23044 tablet_service.cc:1514] Processing DeleteTablet for tablet 3189b8306686492598915de980f225fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:51768
I20260929 16:02:44.960075 23289 tablet_replica.cc:330] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: stopping tablet replica
I20260929 16:02:44.960394 23178 tablet_service.cc:1514] Processing DeleteTablet for tablet 3189b8306686492598915de980f225fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:44 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52586
I20260929 16:02:44.960512 23289 raft_consensus.cc:2229] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:44.961335 23289 raft_consensus.cc:2258] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:44.962139 23290 tablet_replica.cc:330] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: stopping tablet replica
I20260929 16:02:44.962566 23290 raft_consensus.cc:2229] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:44.963115 23290 raft_consensus.cc:2258] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:44.963385 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22847
I20260929 16:02:44.964303 23289 ts_tablet_manager.cc:1905] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:44.965229 23290 ts_tablet_manager.cc:1905] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:44.972767 23290 ts_tablet_manager.cc:1918] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:44.973006 23290 log.cc:1196] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/3189b8306686492598915de980f225fc
I20260929 16:02:44.973664 23290 ts_tablet_manager.cc:1939] T 3189b8306686492598915de980f225fc P b80f60d5141741febc932147796e3cf0: Deleting consensus metadata
I20260929 16:02:44.974005 23289 ts_tablet_manager.cc:1918] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:44.974272 23289 log.cc:1196] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/3189b8306686492598915de980f225fc
I20260929 16:02:44.974982 23289 ts_tablet_manager.cc:1939] T 3189b8306686492598915de980f225fc P 1db2fcd9e1cd4483860e4dae3f3433aa: Deleting consensus metadata
I20260929 16:02:44.975653 22770 catalog_manager.cc:4841] TS b80f60d5141741febc932147796e3cf0 (127.5.117.2:41409): tablet 3189b8306686492598915de980f225fc (table test-table [id=608cbe95128a4e08a09638071dae236e]) successfully deleted
I20260929 16:02:44.976991 22771 catalog_manager.cc:4841] TS 1db2fcd9e1cd4483860e4dae3f3433aa (127.5.117.1:42873): tablet 3189b8306686492598915de980f225fc (table test-table [id=608cbe95128a4e08a09638071dae236e]) successfully deleted
I20260929 16:02:44.977211 23291 tablet_replica.cc:330] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: stopping tablet replica
I20260929 16:02:44.977656 23291 raft_consensus.cc:2229] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:44.978180 23291 raft_consensus.cc:2258] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:44.980423 23291 ts_tablet_manager.cc:1905] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:02:44 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:02:44.988067 23291 ts_tablet_manager.cc:1918] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:44.988296 23291 log.cc:1196] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/3189b8306686492598915de980f225fc
I20260929 16:02:44.988883 23291 ts_tablet_manager.cc:1939] T 3189b8306686492598915de980f225fc P 9d553c6107c4446b82ef6957c3d0136f: Deleting consensus metadata
I20260929 16:02:44.991048 22771 catalog_manager.cc:4841] TS 9d553c6107c4446b82ef6957c3d0136f (127.5.117.3:36835): tablet 3189b8306686492598915de980f225fc (table test-table [id=608cbe95128a4e08a09638071dae236e]) successfully deleted
I20260929 16:02:44.995048 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.966617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.5.117.1:42873)
0929 16:02:44.966831 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.966838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.976519 (+ 9681us) server_negotiation.cc:407] Connection header received
0929 16:02:44.976728 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.976735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.976826 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.976929 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.978830 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.979629 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.981119 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.981658 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.981822 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.986781 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.986814 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.986836 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:44.986907 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.991391 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.992668 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.992676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.992679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.992843 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:44.993297 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:44.993302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:44.993304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:44.993742 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.994159 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.994376 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.994651 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:02:44.995960 22970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:45.099357 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22847
I20260929 16:02:45.121451 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22981
Sep 29 16:02:45 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:02:45.140612 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:45.122285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.5.117.2:41409)
0929 16:02:45.122472 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:45.122476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:45.122494 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:02:45.122658 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:45.122663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:45.122733 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:45.122822 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:45.124137 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:45.124850 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:45.126180 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:45.126586 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:45.126680 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:45.131268 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:45.131283 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:45.131301 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:45.131360 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:45.136454 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:45.137398 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:45.137410 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:45.137414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:45.137597 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:45.138102 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:45.138107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:45.138111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:45.138740 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:45.139120 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:45.139719 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 16:02:45.140187 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:02:45.141419 23104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:45.242919 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22981
I20260929 16:02:45.263036 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23115
Sep 29 16:02:45 dist-test-slave-b69v krb5kdc[22741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:02:45.279703 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:45.264014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.5.117.3:36835)
0929 16:02:45.264271 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:45.264275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:45.264296 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:02:45.264483 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:45.264487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:45.264598 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:45.264697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:45.266311 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:45.267092 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:45.268602 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:45.269087 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:45.269195 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:45.274035 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:45.274050 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:45.274071 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:45.274109 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:45.277197 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:45.277845 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:45.277852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:45.277855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:45.277976 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:45.278284 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:45.278287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:45.278289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:45.278701 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:45.278923 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:45.279149 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 16:02:45.279309 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 16:02:45.280460 23238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:45.375376 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23115
I20260929 16:02:45.394409 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 22754
I20260929 16:02:45.395764 22816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:45.526356 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 22754
2026-09-29T16:02:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4585 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:45Z Disabled control of system clock
I20260929 16:02:45.624354 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34997
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:34997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:46.007587 23300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:46.007894 23300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:46.007995 23300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:46.014989 23300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:46.015087 23300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:46.015125 23300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:46.015180 23300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:46.015224 23300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:46.023921 23300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:34997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:46.025377 23300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:46.027377 23300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:46.040946 23305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:46.040890 23306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:46.040972 23308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:46.042232 23300 server_base.cc:1034] running on GCE node
I20260929 16:02:46.043112 23300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:46.044610 23300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:46.045873 23300 hybrid_clock.cc:648] HybridClock initialized: now 1790697766045842 us; error 32 us; skew 500 ppm
I20260929 16:02:46.046235 23300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:46.048691 23300 webserver.cc:460] Webserver started at http://127.5.117.62:36605/ using document root <none> and password file <none>
I20260929 16:02:46.049376 23300 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:46.049505 23300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:46.049837 23300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:46.053169 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "be8d49e9b9b548e485d2250165ebc9fd"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "33484cdbd9cf377bc52d2d7ee64b791b"
server_key_iv: "8093782b2a7817f6cc996b35ba656b3a"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.054437 23300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "be8d49e9b9b548e485d2250165ebc9fd"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "33484cdbd9cf377bc52d2d7ee64b791b"
server_key_iv: "8093782b2a7817f6cc996b35ba656b3a"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.061272 23300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:02:46.064572 23314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:46.065937 23300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:02:46.066128 23300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "be8d49e9b9b548e485d2250165ebc9fd"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "33484cdbd9cf377bc52d2d7ee64b791b"
server_key_iv: "8093782b2a7817f6cc996b35ba656b3a"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.066341 23300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:46.082283 23300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:46.083091 23300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:46.083370 23300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:46.107125 23300 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:34997
I20260929 16:02:46.107127 23365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:34997 every 8 connection(s)
I20260929 16:02:46.108899 23300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:46.113948 23366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:46.116344 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23300
I20260929 16:02:46.116568 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:02:46.117235 5588 external_mini_cluster.cc:1428] Setting key 196266f1f3e51d51ef070754cc615331
I20260929 16:02:46.127872 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: Bootstrap starting.
I20260929 16:02:46.131865 23366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:46.133322 23366 log.cc:824] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:46.134298 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:46.119613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.5.117.62:34997)
0929 16:02:46.120880 (+ 1267us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:46.120895 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:46.120948 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:02:46.122865 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:46.122891 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:46.123424 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:46.123880 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:46.124605 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.126422 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.128708 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.129586 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.129849 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:46.131119 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:46.131127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:46.131145 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:46.131794 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:46.131910 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:46.131934 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:46.132109 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:46.132400 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 16:02:46.132779 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":487,"threads_started":1}
I20260929 16:02:46.136945 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: No bootstrap required, opened a new log
I20260929 16:02:46.145339 23366 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } }
I20260929 16:02:46.145728 23366 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:46.145849 23366 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8d49e9b9b548e485d2250165ebc9fd, State: Initialized, Role: FOLLOWER
I20260929 16:02:46.146472 23366 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } }
I20260929 16:02:46.146733 23366 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:46.146880 23366 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:46.147075 23366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:46.149399 23366 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } }
I20260929 16:02:46.149921 23366 leader_election.cc:302] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8d49e9b9b548e485d2250165ebc9fd; no voters:
I20260929 16:02:46.150540 23366 leader_election.cc:288] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:46.150682 23371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:46.152019 23371 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [term 1 LEADER]: Becoming Leader. State: Replica: be8d49e9b9b548e485d2250165ebc9fd, State: Running, Role: LEADER
I20260929 16:02:46.152642 23371 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } }
I20260929 16:02:46.152845 23366 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:46.155639 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8d49e9b9b548e485d2250165ebc9fd. Latest consensus state: current_term: 1 leader_uuid: "be8d49e9b9b548e485d2250165ebc9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } } }
I20260929 16:02:46.156301 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:46.155611 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8d49e9b9b548e485d2250165ebc9fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8d49e9b9b548e485d2250165ebc9fd" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34997 } } }
I20260929 16:02:46.156589 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:46.158793 23379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:46.164562 23379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:46.174160 23379 catalog_manager.cc:1347] Generated new cluster ID: 6dfa3405710b445898c172a4416c001a
I20260929 16:02:46.174268 23379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:46.209863 23379 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:46.211318 23379 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:46.220561 23379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: Generated new TSK 0
I20260929 16:02:46.221585 23379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:46.227766 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--builtin_ntp_servers=127.5.117.20:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:46.613128 23390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:46.613432 23390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:46.613523 23390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:46.620298 23390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:46.620379 23390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:46.620482 23390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:46.628629 23390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:46.629818 23390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:46.631585 23390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:46.645234 23396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:46.645293 23395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:46.645493 23398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:46.646078 23390 server_base.cc:1034] running on GCE node
I20260929 16:02:46.646679 23390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:46.647627 23390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:46.648846 23390 hybrid_clock.cc:648] HybridClock initialized: now 1790697766648814 us; error 43 us; skew 500 ppm
I20260929 16:02:46.649188 23390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:46.651443 23390 webserver.cc:460] Webserver started at http://127.5.117.1:36813/ using document root <none> and password file <none>
I20260929 16:02:46.652201 23390 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:46.652344 23390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:46.652683 23390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:46.655615 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "e85895940ef14d1fa7710e2e5f39d4e7"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "e5991d8fc2e0183e1b293d2e04ca5e5c"
server_key_iv: "265886a818398bbb8a683b95fae85e7d"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.656495 23390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "e85895940ef14d1fa7710e2e5f39d4e7"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "e5991d8fc2e0183e1b293d2e04ca5e5c"
server_key_iv: "265886a818398bbb8a683b95fae85e7d"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.661489 23390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:02:46.664812 23404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:46.666114 23390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:46.666282 23390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "e85895940ef14d1fa7710e2e5f39d4e7"
format_stamp: "Formatted at 2026-09-29 16:02:46 on dist-test-slave-b69v"
server_key: "e5991d8fc2e0183e1b293d2e04ca5e5c"
server_key_iv: "265886a818398bbb8a683b95fae85e7d"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.666497 23390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:46.683923 23390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:46.684667 23390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:46.684978 23390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:46.685953 23390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:46.687420 23390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:46.687531 23390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:46.687650 23390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:46.687767 23390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:46.720835 23390 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:40357
I20260929 16:02:46.720841 23516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:40357 every 8 connection(s)
I20260929 16:02:46.722080 23390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:46.728621 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23390
I20260929 16:02:46.728827 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:46.729470 5588 external_mini_cluster.cc:1428] Setting key cfb337a5e8ca3214310317042ee07476
I20260929 16:02:46.732745 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--builtin_ntp_servers=127.5.117.20:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:46.735826 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:46.724713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:38363 (local address 127.5.117.62:34997)
0929 16:02:46.725204 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:46.725211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:46.726316 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 16:02:46.728129 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:46.728133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:46.728215 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:46.728315 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:46.729734 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.730609 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.732572 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.733255 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.733427 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:46.734829 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:46.734837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:46.734844 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:46.734869 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:46.734927 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:46.734949 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:46.735065 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:46.735252 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 16:02:46.735370 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260929 16:02:46.736589 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:46.725268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34997 (local address 127.5.117.1:38363)
0929 16:02:46.726167 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:46.726202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:46.727856 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:46.728598 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:46.728613 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:46.728721 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:46.729556 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:46.729582 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.730784 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.730789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:46.732427 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:46.732440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:46.733495 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:46.733501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:46.733707 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:46.734510 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:46.734513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:46.734549 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:46.734628 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:46.735062 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:46.735070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:46.735075 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:02:46.735209 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 16:02:46.735492 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":283,"threads_started":1}
I20260929 16:02:46.738145 23517 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34997
I20260929 16:02:46.738456 23517 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:46.739173 23517 heartbeater.cc:511] Master 127.5.117.62:34997 requested a full tablet report, sending...
I20260929 16:02:46.741190 23331 ts_manager.cc:194] Registered new tserver with Master: e85895940ef14d1fa7710e2e5f39d4e7 (127.5.117.1:40357)
I20260929 16:02:46.742775 23331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.1:38363
W20260929 16:02:47.095628 23521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:47.096000 23521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:47.096185 23521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:47.103464 23521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:47.103619 23521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:47.103802 23521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:47.113473 23521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:47.115015 23521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:47.117067 23521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:47.131748 23529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.131786 23527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.131829 23526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:47.132555 23521 server_base.cc:1034] running on GCE node
I20260929 16:02:47.133417 23521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:47.134618 23521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:47.136034 23521 hybrid_clock.cc:648] HybridClock initialized: now 1790697767135814 us; error 205 us; skew 500 ppm
I20260929 16:02:47.136462 23521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:47.139075 23521 webserver.cc:460] Webserver started at http://127.5.117.2:45845/ using document root <none> and password file <none>
I20260929 16:02:47.139884 23521 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:47.140026 23521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:47.140437 23521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:47.143621 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "0677702e56ef4de5b8b30554bddba31d"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "11e5d80bc2d493719fc54f57c1bb0e64"
server_key_iv: "9e859e2ff2651335404994322cdb8f0c"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.144608 23521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "0677702e56ef4de5b8b30554bddba31d"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "11e5d80bc2d493719fc54f57c1bb0e64"
server_key_iv: "9e859e2ff2651335404994322cdb8f0c"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.150118 23521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:02:47.154112 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.155619 23521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:02:47.155800 23521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "0677702e56ef4de5b8b30554bddba31d"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "11e5d80bc2d493719fc54f57c1bb0e64"
server_key_iv: "9e859e2ff2651335404994322cdb8f0c"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.156037 23521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:47.171031 23521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:47.171819 23521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:47.172139 23521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:47.173169 23521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:47.175433 23521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:47.175570 23521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.175709 23521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:47.175817 23521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.212865 23521 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:34697
I20260929 16:02:47.212915 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:34697 every 8 connection(s)
I20260929 16:02:47.214367 23521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:47.216213 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23521
I20260929 16:02:47.216428 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:47.217063 5588 external_mini_cluster.cc:1428] Setting key 3bcff221e8feb95bb5ef657deb91244e
I20260929 16:02:47.220732 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--builtin_ntp_servers=127.5.117.20:36767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:47.230170 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.217761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:50105 (local address 127.5.117.62:34997)
0929 16:02:47.218034 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.218049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.219729 (+ 1680us) server_negotiation.cc:407] Connection header received
0929 16:02:47.222070 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.222081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.222237 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.222425 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:47.223658 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.225016 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.226647 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.227388 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.227538 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.229032 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:47.229041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:47.229049 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:47.229077 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:47.229126 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:47.229146 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:47.229366 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.229597 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.229726 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:02:47.231029 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.218349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34997 (local address 127.5.117.2:50105)
0929 16:02:47.219551 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.219601 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.221653 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.222685 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.222694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.222787 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:47.223450 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.223467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.225191 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.225195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.226486 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.226498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.227625 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.227630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.227842 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.228701 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:47.228704 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:47.228745 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:47.228824 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:47.229381 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:47.229390 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:47.229397 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.229553 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.229845 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":354,"threads_started":1}
I20260929 16:02:47.232666 23648 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34997
I20260929 16:02:47.232990 23648 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:47.233865 23648 heartbeater.cc:511] Master 127.5.117.62:34997 requested a full tablet report, sending...
I20260929 16:02:47.235707 23331 ts_manager.cc:194] Registered new tserver with Master: 0677702e56ef4de5b8b30554bddba31d (127.5.117.2:34697)
I20260929 16:02:47.236927 23331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.2:50105
W20260929 16:02:47.593019 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:47.593436 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:47.593621 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:47.601557 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:47.601734 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:47.601938 23651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:47.611995 23651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:34997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:47.613767 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:47.615903 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:47.630897 23657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.630884 23656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.630941 23659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:47.631690 23651 server_base.cc:1034] running on GCE node
I20260929 16:02:47.632462 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:47.633553 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:47.634770 23651 hybrid_clock.cc:648] HybridClock initialized: now 1790697767634718 us; error 41 us; skew 500 ppm
I20260929 16:02:47.635155 23651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:47.637769 23651 webserver.cc:460] Webserver started at http://127.5.117.3:38475/ using document root <none> and password file <none>
I20260929 16:02:47.638600 23651 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:47.638696 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:47.639099 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:47.642505 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "b57cfb6744a94891aaa3f50c885e3e7f"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "31f043d19d1beaea6c406ab462a3f66e"
server_key_iv: "18af0d947ed4a69b78fbfd6c958fed75"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.643417 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "b57cfb6744a94891aaa3f50c885e3e7f"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "31f043d19d1beaea6c406ab462a3f66e"
server_key_iv: "18af0d947ed4a69b78fbfd6c958fed75"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.649464 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:02:47.653481 23665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.655112 23651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:47.655308 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "b57cfb6744a94891aaa3f50c885e3e7f"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-b69v"
server_key: "31f043d19d1beaea6c406ab462a3f66e"
server_key_iv: "18af0d947ed4a69b78fbfd6c958fed75"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.655575 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:47.674656 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:47.675757 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:47.676175 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:47.677409 23651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:47.679117 23651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:47.679260 23651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.679399 23651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:47.679481 23651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.715665 23651 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:38897
I20260929 16:02:47.715710 23777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:38897 every 8 connection(s)
I20260929 16:02:47.717355 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:47.727303 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23651
I20260929 16:02:47.727550 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:47.728245 5588 external_mini_cluster.cc:1428] Setting key 1bda69fbb731c0c0466a409e4889dc44
I20260929 16:02:47.733645 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.720517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:39439 (local address 127.5.117.62:34997)
0929 16:02:47.720786 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.720793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.722464 (+ 1671us) server_negotiation.cc:407] Connection header received
0929 16:02:47.724618 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.724624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.724739 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.724888 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:47.726668 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.727927 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.730272 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.730862 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.730969 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.732848 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:47.732854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:47.732858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:47.732879 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:47.732917 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:47.732930 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:47.733018 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.733249 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.733349 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:02:47.734947 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.721098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:34997 (local address 127.5.117.3:39439)
0929 16:02:47.722256 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.722304 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.724261 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.725361 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.725379 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.725543 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:47.726401 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.726426 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.728222 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.728228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.730098 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.730117 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.731129 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.731137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.731478 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.732486 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:47.732489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:47.732541 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:47.732642 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:47.733023 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:47.733031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:47.733038 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.733209 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.733598 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":390,"threads_started":1}
I20260929 16:02:47.737036 23778 heartbeater.cc:347] Connected to a master server at 127.5.117.62:34997
I20260929 16:02:47.737385 23778 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:47.738320 23778 heartbeater.cc:511] Master 127.5.117.62:34997 requested a full tablet report, sending...
I20260929 16:02:47.740422 23331 ts_manager.cc:194] Registered new tserver with Master: b57cfb6744a94891aaa3f50c885e3e7f (127.5.117.3:38897)
I20260929 16:02:47.741377 23331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.3:39439
I20260929 16:02:47.745028 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:47.746560 23517 heartbeater.cc:503] Master 127.5.117.62:34997 was elected leader, sending a full tablet report...
I20260929 16:02:47.757406 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.750290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.5.117.62:34997)
0929 16:02:47.750590 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.750601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.751060 (+ 459us) server_negotiation.cc:407] Connection header received
0929 16:02:47.751180 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.751184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.751268 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.751388 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:47.751914 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.752796 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.754570 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.755048 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.755162 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.756575 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:47.756580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:47.756584 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:47.756604 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:47.756641 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:47.756654 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:47.756737 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.756905 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.757123 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:02:47.762564 23331 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:47.770372 23331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:47.820420 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.785764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49012)
0929 16:02:47.810254 (+ 24490us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.810284 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.810440 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.812202 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.812212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.812252 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:47.812956 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.812974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.815114 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.815119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.817600 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.817619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.819201 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.819208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.819420 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.819473 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.819633 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.819856 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24318,"mutex_wait_us":118,"thread_start_us":22598,"threads_started":1}
I20260929 16:02:47.821214 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.785466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:38897 (local address 127.0.0.1:38970)
0929 16:02:47.809066 (+ 23600us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.809158 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.809480 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.812253 (+ 2773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.812257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.812274 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:47.813129 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.813144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.815027 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.815033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.817900 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.817915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.820325 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.820332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.820546 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.820578 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.820663 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.820802 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23109,"mutex_wait_us":196,"thread_start_us":27130,"threads_started":1}
I20260929 16:02:47.821223 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.786625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.5.117.1:40357)
0929 16:02:47.810179 (+ 23554us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.810205 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.811021 (+ 816us) server_negotiation.cc:407] Connection header received
0929 16:02:47.811179 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.811189 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.811563 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.811780 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:47.813244 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.814904 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.817838 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.818972 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.819166 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.819547 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.819720 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.819931 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23256,"thread_start_us":23577,"threads_started":1}
I20260929 16:02:47.822676 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.787067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.5.117.2:34697)
0929 16:02:47.807691 (+ 20624us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.807711 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.813097 (+ 5386us) server_negotiation.cc:407] Connection header received
0929 16:02:47.813353 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.813366 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.813665 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.813904 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:47.814499 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.816665 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.820074 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.821656 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.821817 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.822025 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.822139 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.822324 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20293,"thread_start_us":20666,"threads_started":1}
I20260929 16:02:47.823370 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.785882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.5.117.3:38897)
0929 16:02:47.811065 (+ 25183us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.811073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.811106 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:02:47.811198 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.811204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.811524 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.811779 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:47.813342 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.814826 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.818184 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.819671 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.819793 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.820114 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.822873 (+ 2759us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.823069 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24991,"thread_start_us":24786,"threads_started":1}
I20260929 16:02:47.822396 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.786341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:34697 (local address 127.0.0.1:33750)
0929 16:02:47.812919 (+ 26578us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.812942 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.813121 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.813910 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.813914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.813942 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:47.814291 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.814302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.816830 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.816835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.819144 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.819159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.821853 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.821858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.821983 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.822007 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.822083 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.822178 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26419,"thread_start_us":24191,"threads_started":1}
I20260929 16:02:47.826179 23452 tablet_service.cc:1467] Processing CreateTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 (DEFAULT_TABLE table=test-table [id=ae050780df314537814f1f36b42ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:47.827440 23583 tablet_service.cc:1467] Processing CreateTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 (DEFAULT_TABLE table=test-table [id=ae050780df314537814f1f36b42ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:47.827466 23713 tablet_service.cc:1467] Processing CreateTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 (DEFAULT_TABLE table=test-table [id=ae050780df314537814f1f36b42ba4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:47.828119 23452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e546f0b1994c04a6b84b3d4e068de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:47.828688 23583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e546f0b1994c04a6b84b3d4e068de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:47.828925 23713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22e546f0b1994c04a6b84b3d4e068de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:47.841899 23796 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Bootstrap starting.
I20260929 16:02:47.844861 23797 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Bootstrap starting.
I20260929 16:02:47.846957 23796 tablet_bootstrap.cc:654] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:47.848649 23797 tablet_bootstrap.cc:654] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:47.848708 23796 log.cc:824] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:47.849870 23797 log.cc:824] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:47.849996 23798 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Bootstrap starting.
I20260929 16:02:47.852905 23796 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: No bootstrap required, opened a new log
I20260929 16:02:47.853189 23797 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: No bootstrap required, opened a new log
I20260929 16:02:47.853199 23796 ts_tablet_manager.cc:1397] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260929 16:02:47.853482 23797 ts_tablet_manager.cc:1397] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:02:47.853989 23798 tablet_bootstrap.cc:654] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:47.855198 23798 log.cc:824] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:47.858940 23798 tablet_bootstrap.cc:492] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: No bootstrap required, opened a new log
I20260929 16:02:47.859266 23798 ts_tablet_manager.cc:1397] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260929 16:02:47.862788 23796 raft_consensus.cc:348] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.863211 23796 raft_consensus.cc:374] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:47.863324 23796 raft_consensus.cc:729] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0677702e56ef4de5b8b30554bddba31d, State: Initialized, Role: FOLLOWER
I20260929 16:02:47.864003 23796 consensus_queue.cc:260] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.864302 23797 raft_consensus.cc:348] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.864799 23797 raft_consensus.cc:374] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:47.864943 23797 raft_consensus.cc:729] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85895940ef14d1fa7710e2e5f39d4e7, State: Initialized, Role: FOLLOWER
I20260929 16:02:47.865320 23648 heartbeater.cc:503] Master 127.5.117.62:34997 was elected leader, sending a full tablet report...
I20260929 16:02:47.866003 23796 ts_tablet_manager.cc:1428] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:02:47.867506 23797 consensus_queue.cc:260] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.869546 23797 ts_tablet_manager.cc:1428] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260929 16:02:47.870332 23798 raft_consensus.cc:348] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.870777 23798 raft_consensus.cc:374] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:47.870895 23798 raft_consensus.cc:729] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57cfb6744a94891aaa3f50c885e3e7f, State: Initialized, Role: FOLLOWER
I20260929 16:02:47.871634 23798 consensus_queue.cc:260] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.873092 23778 heartbeater.cc:503] Master 127.5.117.62:34997 was elected leader, sending a full tablet report...
I20260929 16:02:47.873698 23798 ts_tablet_manager.cc:1428] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260929 16:02:47.888859 23804 raft_consensus.cc:482] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:47.889096 23804 raft_consensus.cc:504] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.890926 23804 leader_election.cc:288] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e85895940ef14d1fa7710e2e5f39d4e7 (127.5.117.1:40357), 0677702e56ef4de5b8b30554bddba31d (127.5.117.2:34697)
I20260929 16:02:47.898300 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.891531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:53965 (local address 127.5.117.2:34697)
0929 16:02:47.891734 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.891741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.891762 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:02:47.891933 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.891939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.892038 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.892182 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:47.892725 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.893771 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.896298 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.897445 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.897582 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.897688 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.897872 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.898024 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:02:47.898320 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.891252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:34697 (local address 127.5.117.3:53965)
0929 16:02:47.891581 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.891606 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.891742 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.892246 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.892252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.892290 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:47.892547 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.892564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.893908 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.893911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.896152 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.896165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.897601 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.897604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.897723 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.897748 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.897817 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.897926 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":30}
I20260929 16:02:47.898850 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.891307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.5.117.3:54997)
0929 16:02:47.892246 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:47.892263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:47.892492 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:47.892869 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:47.892871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:47.892891 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:47.893196 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.893203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.894366 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.894371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.896435 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:47.896444 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.898075 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.898083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:47.898257 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.898285 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:02:47.898385 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:02:47.898538 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":54,"spinlock_wait_cycles":24960,"thread_start_us":522,"threads_started":1}
I20260929 16:02:47.899201 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.891558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:54997 (local address 127.5.117.1:40357)
0929 16:02:47.891955 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.891961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.892386 (+ 425us) server_negotiation.cc:407] Connection header received
0929 16:02:47.892623 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.892626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.892741 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.892854 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:47.893362 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.894227 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.896615 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.897888 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.898030 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.898133 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.898766 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.898920 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260929 16:02:47.899490 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e546f0b1994c04a6b84b3d4e068de6" candidate_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0677702e56ef4de5b8b30554bddba31d" is_pre_election: true
I20260929 16:02:47.900113 23603 raft_consensus.cc:2454] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57cfb6744a94891aaa3f50c885e3e7f in term 0.
I20260929 16:02:47.900208 23472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e546f0b1994c04a6b84b3d4e068de6" candidate_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" is_pre_election: true
I20260929 16:02:47.900817 23472 raft_consensus.cc:2454] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b57cfb6744a94891aaa3f50c885e3e7f in term 0.
I20260929 16:02:47.900820 23667 leader_election.cc:302] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0677702e56ef4de5b8b30554bddba31d, b57cfb6744a94891aaa3f50c885e3e7f; no voters:
I20260929 16:02:47.901232 23804 raft_consensus.cc:2781] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:47.901422 23804 raft_consensus.cc:482] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:47.901552 23804 raft_consensus.cc:3037] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:47.904158 23804 raft_consensus.cc:504] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.904824 23804 leader_election.cc:288] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [CANDIDATE]: Term 1 election: Requested vote from peers e85895940ef14d1fa7710e2e5f39d4e7 (127.5.117.1:40357), 0677702e56ef4de5b8b30554bddba31d (127.5.117.2:34697)
I20260929 16:02:47.905274 23472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e546f0b1994c04a6b84b3d4e068de6" candidate_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85895940ef14d1fa7710e2e5f39d4e7"
I20260929 16:02:47.905432 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22e546f0b1994c04a6b84b3d4e068de6" candidate_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0677702e56ef4de5b8b30554bddba31d"
I20260929 16:02:47.905535 23472 raft_consensus.cc:3037] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:47.905680 23603 raft_consensus.cc:3037] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:47.909269 23472 raft_consensus.cc:2454] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57cfb6744a94891aaa3f50c885e3e7f in term 1.
I20260929 16:02:47.909269 23603 raft_consensus.cc:2454] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57cfb6744a94891aaa3f50c885e3e7f in term 1.
I20260929 16:02:47.909765 23668 leader_election.cc:302] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b57cfb6744a94891aaa3f50c885e3e7f, e85895940ef14d1fa7710e2e5f39d4e7; no voters:
I20260929 16:02:47.910127 23804 raft_consensus.cc:2781] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:47.910633 23804 raft_consensus.cc:686] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 1 LEADER]: Becoming Leader. State: Replica: b57cfb6744a94891aaa3f50c885e3e7f, State: Running, Role: LEADER
I20260929 16:02:47.911198 23804 consensus_queue.cc:237] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } }
I20260929 16:02:47.916584 23331 catalog_manager.cc:5495] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to b57cfb6744a94891aaa3f50c885e3e7f (127.5.117.3). New cstate: current_term: 1 leader_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57cfb6744a94891aaa3f50c885e3e7f" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 38897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 } health_report { overall_health: UNKNOWN } } }
I20260929 16:02:47.955428 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.948993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.5.117.3:38897)
0929 16:02:47.949203 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:47.949209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:47.949228 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:02:47.949347 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:47.949351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:47.949475 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:47.949622 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:47.949967 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.950748 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.952457 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:47.953107 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:47.953220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:47.953951 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:47.954729 (+ 778us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:47.954870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:47.955000 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 16:02:47.955167 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:02:47.966096 23603 raft_consensus.cc:1261] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 1 FOLLOWER]: Refusing update from remote peer b57cfb6744a94891aaa3f50c885e3e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:47.966096 23472 raft_consensus.cc:1261] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 1 FOLLOWER]: Refusing update from remote peer b57cfb6744a94891aaa3f50c885e3e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 16:02:47.966773 23649 tablet.cc:2369] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:47.967211 23806 consensus_queue.cc:1035] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0677702e56ef4de5b8b30554bddba31d" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:47.967621 23804 consensus_queue.cc:1035] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e85895940ef14d1fa7710e2e5f39d4e7" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:02:47.969266 23779 tablet.cc:2369] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:47.974622 23518 tablet.cc:2369] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:47.979704 23809 mvcc.cc:205] Tried to move back new op lower bound from 7334698057577582592 to 7334698057369120768. Current Snapshot: MvccSnapshot[applied={T|T < 7334698057577582592}]
I20260929 16:02:47.997526 23331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:47.997897 23331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:48.002295 23331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8d49e9b9b548e485d2250165ebc9fd: Sending DeleteTablet for 3 replicas of tablet 22e546f0b1994c04a6b84b3d4e068de6
I20260929 16:02:48.003664 23713 tablet_service.cc:1514] Processing DeleteTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:47 UTC) from {username='slave'} at 127.0.0.1:38970
I20260929 16:02:48.003786 23452 tablet_service.cc:1514] Processing DeleteTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:47 UTC) from {username='slave'} at 127.0.0.1:49012
I20260929 16:02:48.004693 23583 tablet_service.cc:1514] Processing DeleteTablet for tablet 22e546f0b1994c04a6b84b3d4e068de6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:47 UTC) from {username='slave'} at 127.0.0.1:33750
I20260929 16:02:48.005676 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23390
I20260929 16:02:48.019227 23820 tablet_replica.cc:330] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: stopping tablet replica
I20260929 16:02:48.019903 23820 raft_consensus.cc:2229] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:48.020859 23820 raft_consensus.cc:2258] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:48.022065 23821 tablet_replica.cc:330] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: stopping tablet replica
I20260929 16:02:48.022691 23821 raft_consensus.cc:2229] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:48.023371 23821 raft_consensus.cc:2258] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:48.026466 23821 ts_tablet_manager.cc:1905] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:48.026463 23820 ts_tablet_manager.cc:1905] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:48.028528 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.007587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.5.117.1:40357)
0929 16:02:48.007849 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:48.007859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:48.018577 (+ 10718us) server_negotiation.cc:407] Connection header received
0929 16:02:48.019207 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:48.019218 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:48.019376 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:48.019471 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:48.020518 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.021791 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.023777 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.024501 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.024630 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:48.026419 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:48.026431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:48.026455 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:48.027365 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:48.027438 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:48.027487 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:48.027581 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:48.027850 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:02:48.028016 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:02:48.028939 23819 tablet_replica.cc:330] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7: stopping tablet replica
I20260929 16:02:48.029552 23819 raft_consensus.cc:2229] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:48.030238 23819 raft_consensus.cc:2258] T 22e546f0b1994c04a6b84b3d4e068de6 P e85895940ef14d1fa7710e2e5f39d4e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:48.030263 23512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:48.034693 23821 ts_tablet_manager.cc:1918] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:48.034962 23821 log.cc:1196] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/22e546f0b1994c04a6b84b3d4e068de6
I20260929 16:02:48.035743 23821 ts_tablet_manager.cc:1939] T 22e546f0b1994c04a6b84b3d4e068de6 P 0677702e56ef4de5b8b30554bddba31d: Deleting consensus metadata
I20260929 16:02:48.037477 23820 ts_tablet_manager.cc:1918] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:48.037719 23820 log.cc:1196] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/22e546f0b1994c04a6b84b3d4e068de6
I20260929 16:02:48.038374 23820 ts_tablet_manager.cc:1939] T 22e546f0b1994c04a6b84b3d4e068de6 P b57cfb6744a94891aaa3f50c885e3e7f: Deleting consensus metadata
I20260929 16:02:48.038520 23316 catalog_manager.cc:4841] TS 0677702e56ef4de5b8b30554bddba31d (127.5.117.2:34697): tablet 22e546f0b1994c04a6b84b3d4e068de6 (table test-table [id=ae050780df314537814f1f36b42ba4fe]) successfully deleted
I20260929 16:02:48.040463 23318 catalog_manager.cc:4841] TS b57cfb6744a94891aaa3f50c885e3e7f (127.5.117.3:38897): tablet 22e546f0b1994c04a6b84b3d4e068de6 (table test-table [id=ae050780df314537814f1f36b42ba4fe]) successfully deleted
I20260929 16:02:48.142287 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23390
W20260929 16:02:48.164837 23317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.117.1:40357 (error 108)
I20260929 16:02:48.165331 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23521
I20260929 16:02:48.167735 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.167055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49036)
0929 16:02:48.167336 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.167515 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260929 16:02:48.168277 23317 catalog_manager.cc:4584] TS e85895940ef14d1fa7710e2e5f39d4e7 (127.5.117.1:40357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22e546f0b1994c04a6b84b3d4e068de6: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
I20260929 16:02:48.175464 23795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.166641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.5.117.2:34697)
0929 16:02:48.166861 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:48.166868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:48.166977 (+ 109us) server_negotiation.cc:407] Connection header received
0929 16:02:48.167287 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:48.167293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:48.167433 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:48.167564 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:48.168252 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.169184 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.171382 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.171927 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.172132 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:48.173639 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:48.173648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:48.173667 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:48.174480 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:48.174557 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:48.174581 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:48.174694 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:48.174873 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:02:48.175011 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:02:48.176400 23643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:48.228446 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.227711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49046)
0929 16:02:48.228079 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.228190 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260929 16:02:48.229859 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.229435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49052)
0929 16:02:48.229637 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.229718 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 16:02:48.291064 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.290296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49054)
0929 16:02:48.290643 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.290762 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260929 16:02:48.292798 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.292264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49058)
0929 16:02:48.292527 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.292627 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 16:02:48.305399 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23521
I20260929 16:02:48.328215 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23651
I20260929 16:02:48.338449 23792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.329332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.5.117.3:38897)
0929 16:02:48.329563 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:48.329570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:48.329598 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:02:48.329808 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:48.329815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:48.329999 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:48.330133 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:48.330672 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.332007 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.334341 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.335238 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.335397 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:48.336794 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:48.336804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:48.336822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:48.337585 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:48.337641 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:48.337663 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:48.337814 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:48.337926 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:02:48.338049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:02:48.339304 23773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:48.374578 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.373902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49070)
0929 16:02:48.374212 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.374359 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 16:02:48.375998 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.375516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:40357 (local address 127.0.0.1:49086)
0929 16:02:48.375769 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:48.375864 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.1:40357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 16:02:48.477929 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23651
I20260929 16:02:48.501641 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23300
I20260929 16:02:48.503048 23361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:48.629346 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23300
2026-09-29T16:02:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3099 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:48 dist-test-slave-b69v krb5kdc[23830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:48 dist-test-slave-b69v krb5kdc[23830](info): set up 2 sockets
Sep 29 16:02:48 dist-test-slave-b69v krb5kdc[23830](info): commencing operation
krb5kdc: starting...
W20260929 16:02:49.788614 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.010s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:02:49 dist-test-slave-b69v krb5kdc[23830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:02:50.092615 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37059
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:37059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:50.475333 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:50.475721 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:50.475967 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:50.484493 23843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:50.484611 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:50.484711 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:50.484807 23843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:50.484894 23843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:50.495411 23843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:37059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:50.497046 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:50.498579 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:50.511704 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:50.512100 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:50.512183 23851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:50.513446 23843 server_base.cc:1034] running on GCE node
I20260929 16:02:50.514313 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:50.515858 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:50.517131 23843 hybrid_clock.cc:648] HybridClock initialized: now 1790697770517080 us; error 37 us; skew 500 ppm
I20260929 16:02:50.517535 23843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:50 dist-test-slave-b69v krb5kdc[23830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697770, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:50.521059 23843 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:02:50.522679 23843 webserver.cc:460] Webserver started at http://127.5.117.62:41199/ using document root <none> and password file <none>
I20260929 16:02:50.523428 23843 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:50.523583 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:50.523933 23843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:50.527282 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-b69v"
server_key: "6edd1627270cbd5b457448e0f7dbc1f9"
server_key_iv: "9425e1de80940a066d9eae1c712fbf90"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.528236 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-b69v"
server_key: "6edd1627270cbd5b457448e0f7dbc1f9"
server_key_iv: "9425e1de80940a066d9eae1c712fbf90"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.534695 23843 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:02:50.538990 23858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:50.540645 23843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:50.540856 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-b69v"
server_key: "6edd1627270cbd5b457448e0f7dbc1f9"
server_key_iv: "9425e1de80940a066d9eae1c712fbf90"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.541103 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:50.555212 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:50.560647 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:50.560969 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:50.582666 23843 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:37059
I20260929 16:02:50.582742 23909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:37059 every 8 connection(s)
I20260929 16:02:50.584770 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:50.589524 23910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:50.593686 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23843
I20260929 16:02:50.593946 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:02:50.594491 5588 external_mini_cluster.cc:1428] Setting key 44f73c0d0d2697716f5e62caddf1ebd3
I20260929 16:02:50.604316 23910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: Bootstrap starting.
I20260929 16:02:50.608726 23910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:50.610208 23910 log.cc:824] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:02:50 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:50.613497 23910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: No bootstrap required, opened a new log
I20260929 16:02:50.620814 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:50.596713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.5.117.62:37059)
0929 16:02:50.597636 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:50.597650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:50.597697 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:02:50.599604 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:50.599631 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:50.600189 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:50.600604 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:50.602573 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.603975 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.606231 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.606980 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.607127 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:50.612983 (+ 5856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:50.612991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:50.613006 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:50.613042 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:50.616475 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:50.617240 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:50.617246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:50.617251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:50.617438 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:50.617840 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:50.617849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:50.617854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:50.618415 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:50.618778 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:50.619215 (+ 437us) server_negotiation.cc:299] Negotiation successful
0929 16:02:50.619659 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":442,"threads_started":1}
I20260929 16:02:50.622054 23910 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } }
I20260929 16:02:50.622474 23910 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:50.622615 23910 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef51ed74d4fa4d3bbbed1e51e6e74c69, State: Initialized, Role: FOLLOWER
I20260929 16:02:50.623270 23910 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } }
I20260929 16:02:50.623549 23910 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:50.623730 23910 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:50.623929 23910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:50.626216 23910 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } }
I20260929 16:02:50.626700 23910 leader_election.cc:302] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef51ed74d4fa4d3bbbed1e51e6e74c69; no voters:
I20260929 16:02:50.627307 23910 leader_election.cc:288] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:50.627450 23915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:50.628314 23915 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [term 1 LEADER]: Becoming Leader. State: Replica: ef51ed74d4fa4d3bbbed1e51e6e74c69, State: Running, Role: LEADER
I20260929 16:02:50.628883 23915 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } }
I20260929 16:02:50.629537 23910 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:50.633706 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef51ed74d4fa4d3bbbed1e51e6e74c69. Latest consensus state: current_term: 1 leader_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } } }
I20260929 16:02:50.634114 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:50.634681 23916 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef51ed74d4fa4d3bbbed1e51e6e74c69" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37059 } } }
I20260929 16:02:50.635026 23916 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:50.639881 23925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:50.645023 23925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:50.646384 23933 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:02:50.654448 23925 catalog_manager.cc:1347] Generated new cluster ID: 15d10e3f2de44b258007d1f20296f300
I20260929 16:02:50.654564 23925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:50.688484 23925 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:50.690658 23925 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:50.697722 23925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: Generated new TSK 0
I20260929 16:02:50.698843 23925 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:02:50.775877 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:51.161090 23937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:51.161407 23937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:51.161589 23937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:51.169201 23937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:51.169304 23937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:51.169533 23937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:51.179777 23937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:51.181249 23937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:51.182767 23937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:51.197645 23945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.197906 23943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.198467 23942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:51.198513 23937 server_base.cc:1034] running on GCE node
I20260929 16:02:51.199270 23937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:51.200294 23937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:51.201658 23937 hybrid_clock.cc:648] HybridClock initialized: now 1790697771201601 us; error 55 us; skew 500 ppm
I20260929 16:02:51.202008 23937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:51 dist-test-slave-b69v krb5kdc[23830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:51.205834 23937 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:02:51.207651 23937 webserver.cc:460] Webserver started at http://127.5.117.1:36353/ using document root <none> and password file <none>
I20260929 16:02:51.208519 23937 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:51.208658 23937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:51.209091 23937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:51.212399 23937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "1df911250338423da7e33a786bba54c4"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "439e12ba6f51e43d29f8b54b0b69593d"
server_key_iv: "185e49e60b0fefe35bccc2f272e60b67"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.213258 23937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "1df911250338423da7e33a786bba54c4"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "439e12ba6f51e43d29f8b54b0b69593d"
server_key_iv: "185e49e60b0fefe35bccc2f272e60b67"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.218757 23937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:02:51.223333 23952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.225216 23937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 16:02:51.225405 23937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "1df911250338423da7e33a786bba54c4"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "439e12ba6f51e43d29f8b54b0b69593d"
server_key_iv: "185e49e60b0fefe35bccc2f272e60b67"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.225617 23937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:51.241731 23937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:51.246618 23937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:51.246935 23937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:51.248342 23937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:51.249957 23937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:51.250034 23937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.250198 23937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:51.250273 23937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.290599 24064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:46475 every 8 connection(s)
I20260929 16:02:51.290656 23937 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:46475
I20260929 16:02:51.292457 23937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:51.301048 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 23937
I20260929 16:02:51.301254 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:51.301769 5588 external_mini_cluster.cc:1428] Setting key 69b43890457bce1703d29f6121437317
Sep 29 16:02:51 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:51.316864 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:51.295793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:47981 (local address 127.5.117.62:37059)
0929 16:02:51.296401 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:51.296407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:51.297395 (+ 988us) server_negotiation.cc:407] Connection header received
0929 16:02:51.299414 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:51.299420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:51.299518 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:51.299652 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:51.301314 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.302193 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.304206 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.304634 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.304781 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:51.308666 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:51.308676 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:51.308683 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:51.308731 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:51.312817 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:51.313654 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:51.313663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:51.313669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:51.313834 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:51.314364 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:51.314371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:51.314377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:51.314795 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:51.314957 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:51.316005 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 16:02:51.316350 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":341,"threads_started":1}
I20260929 16:02:51.317725 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:51.296428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37059 (local address 127.5.117.1:47981)
0929 16:02:51.297219 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:51.297274 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:51.299127 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:51.299902 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:51.299910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:51.300514 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:51.301103 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:51.301149 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.302368 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.302373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:51.303987 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:51.304002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.304881 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.304890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:51.305153 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:51.305906 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:51.305930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:51.308390 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:51.312982 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:51.312990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:51.313007 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:51.313494 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:51.313985 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:51.313988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:51.313991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:51.314211 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:51.315042 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:51.315051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:51.315530 (+ 479us) client_negotiation.cc:769] Sending connection context
0929 16:02:51.315804 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 16:02:51.316355 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":311,"threads_started":1}
I20260929 16:02:51.320032 24065 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37059
I20260929 16:02:51.320493 24065 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:51.321412 24065 heartbeater.cc:511] Master 127.5.117.62:37059 requested a full tablet report, sending...
I20260929 16:02:51.323987 23875 ts_manager.cc:194] Registered new tserver with Master: 1df911250338423da7e33a786bba54c4 (127.5.117.1:46475)
I20260929 16:02:51.325671 23875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:47981
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:02:51.373656 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:51.749987 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:51.750341 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:51.750523 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:51.757954 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:51.758101 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:51.758265 24071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:51.767851 24071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:51.769330 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:51.770742 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:51.784190 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.784300 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.784277 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:51.785398 24071 server_base.cc:1034] running on GCE node
I20260929 16:02:51.785928 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:51.786841 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:51.788036 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790697771787990 us; error 42 us; skew 500 ppm
I20260929 16:02:51.788348 24071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:51 dist-test-slave-b69v krb5kdc[23830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:51.791389 24071 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:02:51.792886 24071 webserver.cc:460] Webserver started at http://127.5.117.2:37037/ using document root <none> and password file <none>
I20260929 16:02:51.793623 24071 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:51.793780 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:51.794135 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:51.797366 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "de17f198d136ff219d37d9a1e9d431eb"
server_key_iv: "09aaf7f262d8033eabb9dd1188fbe823"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.798192 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "de17f198d136ff219d37d9a1e9d431eb"
server_key_iv: "09aaf7f262d8033eabb9dd1188fbe823"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.803396 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:02:51.806864 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.808251 24071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:51.808441 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e"
format_stamp: "Formatted at 2026-09-29 16:02:51 on dist-test-slave-b69v"
server_key: "de17f198d136ff219d37d9a1e9d431eb"
server_key_iv: "09aaf7f262d8033eabb9dd1188fbe823"
server_key_version: "encryptionkey@0"
I20260929 16:02:51.808650 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:51.822829 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:51.826571 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:51.826869 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:51.827834 24071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:51.829200 24071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:51.829269 24071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.829444 24071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:51.829521 24071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:51.868896 24071 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:36821
I20260929 16:02:51.868935 24198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:36821 every 8 connection(s)
I20260929 16:02:51.870416 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:51.876883 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24071
I20260929 16:02:51.877131 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:51.877657 5588 external_mini_cluster.cc:1428] Setting key f43ddbb2fb1cd50bb71df38bc3fe1bc1
Sep 29 16:02:51 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697771, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:51.894899 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:51.873555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:41615 (local address 127.5.117.62:37059)
0929 16:02:51.874083 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:51.874088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:51.874880 (+ 792us) server_negotiation.cc:407] Connection header received
0929 16:02:51.877299 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:51.877311 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:51.877463 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:51.877659 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:51.879863 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.880679 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.882301 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.882709 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.882845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:51.886756 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:51.886766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:51.886774 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:51.886828 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:51.890820 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:51.891668 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:51.891677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:51.891683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:51.891849 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:51.892367 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:51.892375 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:51.892381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:51.892828 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:51.893010 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:51.894313 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0929 16:02:51.894573 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":276,"threads_started":1}
I20260929 16:02:51.896554 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:51.873901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37059 (local address 127.5.117.2:41615)
0929 16:02:51.874700 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:51.874740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:51.876731 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:51.878043 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:51.878055 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:51.878778 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:51.879645 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:51.879686 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.880858 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.880863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:51.882148 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:51.882164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:51.882952 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:51.882959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:51.883190 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:51.883940 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:51.883963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:51.886498 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:51.890995 (+ 4497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:51.891002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:51.891016 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:51.891531 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:51.891976 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:51.891980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:51.891982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:51.892245 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:51.893045 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:51.893052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:51.893698 (+ 646us) client_negotiation.cc:769] Sending connection context
0929 16:02:51.894112 (+ 414us) client_negotiation.cc:240] Negotiation successful
0929 16:02:51.894846 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260929 16:02:51.898447 24199 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37059
I20260929 16:02:51.898829 24199 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:51.899698 24199 heartbeater.cc:511] Master 127.5.117.62:37059 requested a full tablet report, sending...
I20260929 16:02:51.901520 23875 ts_manager.cc:194] Registered new tserver with Master: bbd72bbbbdec4b739e3fc1eff2ea020e (127.5.117.2:36821)
I20260929 16:02:51.902468 23875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:41615
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:02:51.947306 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:52.322932 24205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:52.323253 24205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:52.323357 24205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:02:52.329033 24065 heartbeater.cc:503] Master 127.5.117.62:37059 was elected leader, sending a full tablet report...
W20260929 16:02:52.331408 24205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:52.331581 24205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:52.331796 24205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:52.342911 24205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:52.344636 24205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:52.346306 24205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:52.361788 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:52.362519 24211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:52.363428 24213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:52.363785 24205 server_base.cc:1034] running on GCE node
I20260929 16:02:52.364513 24205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:52.365511 24205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:52.366709 24205 hybrid_clock.cc:648] HybridClock initialized: now 1790697772366674 us; error 38 us; skew 500 ppm
I20260929 16:02:52.367046 24205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:52 dist-test-slave-b69v krb5kdc[23830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697772, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:52.370729 24205 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:02:52.372741 24205 webserver.cc:460] Webserver started at http://127.5.117.3:38429/ using document root <none> and password file <none>
I20260929 16:02:52.373548 24205 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:52.373658 24205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:52.374058 24205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:52.377341 24205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "bde81d48234a4e268049a5934fb286b1"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-b69v"
server_key: "e7208ed6fc1b0ace430b71696529f5a6"
server_key_iv: "79bc661e1818232530439b1ed9682748"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.378252 24205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "bde81d48234a4e268049a5934fb286b1"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-b69v"
server_key: "e7208ed6fc1b0ace430b71696529f5a6"
server_key_iv: "79bc661e1818232530439b1ed9682748"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.383791 24205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:02:52.387478 24220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:52.389384 24205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:52.389578 24205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "bde81d48234a4e268049a5934fb286b1"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-b69v"
server_key: "e7208ed6fc1b0ace430b71696529f5a6"
server_key_iv: "79bc661e1818232530439b1ed9682748"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.389829 24205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:52.404027 24205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:52.408505 24205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:52.408816 24205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:52.409786 24205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:52.411201 24205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:52.411304 24205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:52.411455 24205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:52.411535 24205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:52.448700 24205 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:36213
I20260929 16:02:52.448747 24332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:36213 every 8 connection(s)
I20260929 16:02:52.450320 24205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:52.459389 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24205
I20260929 16:02:52.459601 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:52.460175 5588 external_mini_cluster.cc:1428] Setting key cd0aa4fcd63120e469215b434f03df8c
Sep 29 16:02:52 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697772, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:52.476684 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.454298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:47987 (local address 127.5.117.62:37059)
0929 16:02:52.454990 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.455000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.456339 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 16:02:52.458530 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.458535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.458628 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.458984 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:52.461119 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.462337 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.464756 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.465219 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.465336 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.469487 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:52.469493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:52.469497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:52.469530 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:52.472606 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.473472 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.473477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.473479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.473601 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.474124 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.474128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.474130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.474536 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:52.474680 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.475877 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.476193 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":365,"threads_started":1}
I20260929 16:02:52.477880 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.455069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37059 (local address 127.5.117.3:47987)
0929 16:02:52.456117 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.456184 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.458200 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.459176 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.459185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.459967 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:52.460884 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.460916 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.462588 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.462597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.464575 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.464591 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.465460 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.465466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.465688 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.466474 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:52.466498 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:52.469249 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:52.472783 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:52.472794 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:52.472807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:52.473316 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:52.473740 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:52.473745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:52.473747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:52.473958 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:52.474822 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:52.474831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:52.475327 (+ 496us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.475723 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.476291 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":363,"threads_started":1}
I20260929 16:02:52.479363 24333 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37059
I20260929 16:02:52.479693 24333 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:52.480574 24333 heartbeater.cc:511] Master 127.5.117.62:37059 requested a full tablet report, sending...
I20260929 16:02:52.482292 23875 ts_manager.cc:194] Registered new tserver with Master: bde81d48234a4e268049a5934fb286b1 (127.5.117.3:36213)
I20260929 16:02:52.483317 23875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:47987
I20260929 16:02:52.489610 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:52.510063 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.494569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.5.117.62:37059)
0929 16:02:52.494803 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.494812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.495284 (+ 472us) server_negotiation.cc:407] Connection header received
0929 16:02:52.495458 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.495463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.495527 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.495614 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:52.497467 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.498396 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.499954 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.500484 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.500616 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.503653 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:52.503658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:52.503662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:52.503701 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:52.506640 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.507487 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.507491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.507493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.507610 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.508119 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.508127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.508132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.508504 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:52.508727 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.509258 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.509612 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:02:52.515475 23875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:52.523017 23875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:52.560706 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.537494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.5.117.1:46475)
0929 16:02:52.550682 (+ 13188us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.550692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.550730 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:02:52.551771 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.551779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.552014 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.553026 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:52.553615 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.555346 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.558046 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.559390 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.559565 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.559832 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.560146 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.560325 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12998,"thread_start_us":12793,"threads_started":1}
I20260929 16:02:52.560789 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.537192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:46475 (local address 127.0.0.1:42626)
0929 16:02:52.549562 (+ 12370us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.549615 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.549877 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.552251 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.552255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.552272 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:52.553096 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.553115 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.555499 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.555507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.557830 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.557847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.559629 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.559636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.559832 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.559893 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.560081 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.560275 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12027,"mutex_wait_us":29,"thread_start_us":13772,"threads_started":1}
I20260929 16:02:52.561564 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.537867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.5.117.2:36821)
0929 16:02:52.548616 (+ 10749us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.548630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.550837 (+ 2207us) server_negotiation.cc:407] Connection header received
0929 16:02:52.551027 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.551035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.551307 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.551530 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:52.554073 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.555858 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.558620 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.560117 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.560208 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.560393 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.561040 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.561221 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10502,"thread_start_us":10746,"threads_started":1}
I20260929 16:02:52.564366 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.537481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:36821 (local address 127.0.0.1:38290)
0929 16:02:52.550715 (+ 13234us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.550734 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.550874 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.552202 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.552212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.552250 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:52.553853 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.553869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.555984 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.555989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.558429 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.558446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.560600 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.560606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.560824 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.560862 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.561858 (+ 996us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.561988 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13112,"mutex_wait_us":32,"spinlock_wait_cycles":1408,"thread_start_us":11619,"threads_started":1}
I20260929 16:02:52.565348 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.541841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.5.117.3:36213)
0929 16:02:52.553435 (+ 11594us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.553446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.553584 (+ 138us) server_negotiation.cc:407] Connection header received
0929 16:02:52.553915 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.553923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.554134 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.554346 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:52.556375 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.558069 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.560655 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.562093 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.562212 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.562447 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.564927 (+ 2480us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.565064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11358,"thread_start_us":11164,"threads_started":1}
I20260929 16:02:52.565846 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.541020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:36213 (local address 127.0.0.1:45432)
0929 16:02:52.553462 (+ 12442us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.553482 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.553678 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.554660 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.554665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.554693 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:52.556102 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.556125 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.558237 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.558244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.560464 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.560478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.564315 (+ 3837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.564325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.564498 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.564575 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.564659 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.565551 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12191,"mutex_wait_us":22,"thread_start_us":11833,"threads_started":1}
I20260929 16:02:52.565374 24000 tablet_service.cc:1467] Processing CreateTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 (DEFAULT_TABLE table=test-table [id=26c901f99e7f444596a4471762a6ac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:52.567230 24000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bece0a1a4fa4017ad69b968101bb3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:52.568197 24134 tablet_service.cc:1467] Processing CreateTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 (DEFAULT_TABLE table=test-table [id=26c901f99e7f444596a4471762a6ac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:52.569133 24268 tablet_service.cc:1467] Processing CreateTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 (DEFAULT_TABLE table=test-table [id=26c901f99e7f444596a4471762a6ac4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:52.569636 24134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bece0a1a4fa4017ad69b968101bb3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:52.570217 24268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bece0a1a4fa4017ad69b968101bb3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:52.587603 24354 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Bootstrap starting.
I20260929 16:02:52.592236 24352 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Bootstrap starting.
I20260929 16:02:52.592845 24353 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Bootstrap starting.
I20260929 16:02:52.596721 24353 tablet_bootstrap.cc:654] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:52.598157 24353 log.cc:824] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:52.598333 24354 tablet_bootstrap.cc:654] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:52.599789 24354 log.cc:824] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:52.601109 24352 tablet_bootstrap.cc:654] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:52.602315 24352 log.cc:824] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:52.608417 24353 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: No bootstrap required, opened a new log
I20260929 16:02:52.608841 24353 ts_tablet_manager.cc:1397] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260929 16:02:52.612526 24354 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: No bootstrap required, opened a new log
I20260929 16:02:52.612596 24352 tablet_bootstrap.cc:492] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: No bootstrap required, opened a new log
I20260929 16:02:52.612825 24354 ts_tablet_manager.cc:1397] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.008s
I20260929 16:02:52.612895 24352 ts_tablet_manager.cc:1397] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.000s
I20260929 16:02:52.620227 24353 raft_consensus.cc:348] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.620708 24353 raft_consensus.cc:374] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:52.620534 24354 raft_consensus.cc:348] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.620831 24353 raft_consensus.cc:729] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df911250338423da7e33a786bba54c4, State: Initialized, Role: FOLLOWER
I20260929 16:02:52.620970 24354 raft_consensus.cc:374] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:52.621109 24354 raft_consensus.cc:729] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd72bbbbdec4b739e3fc1eff2ea020e, State: Initialized, Role: FOLLOWER
I20260929 16:02:52.621565 24353 consensus_queue.cc:260] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.621663 24354 consensus_queue.cc:260] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.623392 24199 heartbeater.cc:503] Master 127.5.117.62:37059 was elected leader, sending a full tablet report...
I20260929 16:02:52.623507 24353 ts_tablet_manager.cc:1428] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:02:52.623549 24352 raft_consensus.cc:348] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.623979 24354 ts_tablet_manager.cc:1428] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260929 16:02:52.623970 24352 raft_consensus.cc:374] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:52.624114 24352 raft_consensus.cc:729] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde81d48234a4e268049a5934fb286b1, State: Initialized, Role: FOLLOWER
I20260929 16:02:52.624792 24352 consensus_queue.cc:260] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.626187 24333 heartbeater.cc:503] Master 127.5.117.62:37059 was elected leader, sending a full tablet report...
I20260929 16:02:52.627539 24352 ts_tablet_manager.cc:1428] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260929 16:02:52.702638 24334 tablet.cc:2369] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:52.719817 24359 raft_consensus.cc:482] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:52.720275 24359 raft_consensus.cc:504] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.722285 24359 leader_election.cc:288] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1df911250338423da7e33a786bba54c4 (127.5.117.1:46475), bde81d48234a4e268049a5934fb286b1 (127.5.117.3:36213)
I20260929 16:02:52.732266 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.722927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:40359 (local address 127.5.117.1:46475)
0929 16:02:52.723158 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.723165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.724047 (+ 882us) server_negotiation.cc:407] Connection header received
0929 16:02:52.724307 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.724313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.724398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.724502 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:52.725326 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.726512 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.729647 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.730945 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.731092 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.731186 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.731830 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.731976 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:02:52.732262 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.722917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:57377 (local address 127.5.117.3:36213)
0929 16:02:52.723158 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.723165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.724082 (+ 917us) server_negotiation.cc:407] Connection header received
0929 16:02:52.724307 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.724313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.724398 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.724508 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:52.725324 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.726530 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.729647 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.730907 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.731064 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.731174 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.731830 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.731968 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:02:52.732583 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.722737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:46475 (local address 127.5.117.2:40359)
0929 16:02:52.723886 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.723910 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.724144 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.724530 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.724534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.724574 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:52.725192 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.725202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.726673 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.726678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.729486 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.729501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.731265 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.731273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.731568 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.731673 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.731786 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.731963 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":302,"spinlock_wait_cycles":19328,"thread_start_us":649,"threads_started":1}
I20260929 16:02:52.732609 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.722737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:36213 (local address 127.5.117.2:57377)
0929 16:02:52.723953 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:52.723969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:52.724144 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:52.724529 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:52.724534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:52.724577 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:52.725135 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.725151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.726672 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.726678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.729486 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:52.729501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.731265 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.731273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:52.731560 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.731670 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 16:02:52.731786 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 16:02:52.731963 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"mutex_wait_us":288,"spinlock_wait_cycles":52864,"thread_start_us":660,"threads_started":1}
I20260929 16:02:52.733840 24020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bece0a1a4fa4017ad69b968101bb3d4" candidate_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df911250338423da7e33a786bba54c4" is_pre_election: true
I20260929 16:02:52.733949 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bece0a1a4fa4017ad69b968101bb3d4" candidate_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde81d48234a4e268049a5934fb286b1" is_pre_election: true
I20260929 16:02:52.734498 24020 raft_consensus.cc:2454] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd72bbbbdec4b739e3fc1eff2ea020e in term 0.
I20260929 16:02:52.734496 24288 raft_consensus.cc:2454] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbd72bbbbdec4b739e3fc1eff2ea020e in term 0.
I20260929 16:02:52.735198 24087 leader_election.cc:302] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1df911250338423da7e33a786bba54c4, bbd72bbbbdec4b739e3fc1eff2ea020e, bde81d48234a4e268049a5934fb286b1; no voters:
I20260929 16:02:52.735635 24359 raft_consensus.cc:2781] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:52.735786 24359 raft_consensus.cc:482] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:52.735885 24359 raft_consensus.cc:3037] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:52.738456 24359 raft_consensus.cc:504] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.739141 24359 leader_election.cc:288] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [CANDIDATE]: Term 1 election: Requested vote from peers 1df911250338423da7e33a786bba54c4 (127.5.117.1:46475), bde81d48234a4e268049a5934fb286b1 (127.5.117.3:36213)
I20260929 16:02:52.739497 24020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bece0a1a4fa4017ad69b968101bb3d4" candidate_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1df911250338423da7e33a786bba54c4"
I20260929 16:02:52.739639 24288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bece0a1a4fa4017ad69b968101bb3d4" candidate_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde81d48234a4e268049a5934fb286b1"
I20260929 16:02:52.739737 24020 raft_consensus.cc:3037] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:52.739861 24288 raft_consensus.cc:3037] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:52.743197 24020 raft_consensus.cc:2454] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd72bbbbdec4b739e3fc1eff2ea020e in term 1.
I20260929 16:02:52.743203 24288 raft_consensus.cc:2454] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbd72bbbbdec4b739e3fc1eff2ea020e in term 1.
I20260929 16:02:52.743620 24087 leader_election.cc:302] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1df911250338423da7e33a786bba54c4, bbd72bbbbdec4b739e3fc1eff2ea020e, bde81d48234a4e268049a5934fb286b1; no voters:
I20260929 16:02:52.744221 24359 raft_consensus.cc:2781] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:52.744863 24359 raft_consensus.cc:686] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 1 LEADER]: Becoming Leader. State: Replica: bbd72bbbbdec4b739e3fc1eff2ea020e, State: Running, Role: LEADER
I20260929 16:02:52.745471 24359 consensus_queue.cc:237] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } }
I20260929 16:02:52.750481 23874 catalog_manager.cc:5495] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e reported cstate change: term changed from 0 to 1, leader changed from <none> to bbd72bbbbdec4b739e3fc1eff2ea020e (127.5.117.2). New cstate: current_term: 1 leader_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbd72bbbbdec4b739e3fc1eff2ea020e" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 36821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 } health_report { overall_health: UNKNOWN } } }
W20260929 16:02:52.796041 24066 tablet.cc:2369] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:52.806608 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.799808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.5.117.2:36821)
0929 16:02:52.800037 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.800042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.800114 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:02:52.800286 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.800291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.800473 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.800647 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:52.801263 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.802133 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.803705 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.804214 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.804341 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.805156 (+ 815us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:52.805767 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:52.805911 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.806145 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.806267 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:02:52.816157 24288 raft_consensus.cc:1261] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 1 FOLLOWER]: Refusing update from remote peer bbd72bbbbdec4b739e3fc1eff2ea020e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:52.816169 24020 raft_consensus.cc:1261] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 1 FOLLOWER]: Refusing update from remote peer bbd72bbbbdec4b739e3fc1eff2ea020e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:52.817245 24363 consensus_queue.cc:1035] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde81d48234a4e268049a5934fb286b1" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 36213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:52.817698 24359 consensus_queue.cc:1035] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1df911250338423da7e33a786bba54c4" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 46475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:52.848340 23875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:52.848707 23875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:52.852960 23875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef51ed74d4fa4d3bbbed1e51e6e74c69: Sending DeleteTablet for 3 replicas of tablet 6bece0a1a4fa4017ad69b968101bb3d4
I20260929 16:02:52.854174 24000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:52 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:42626
I20260929 16:02:52.854182 24134 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:52 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:38290
I20260929 16:02:52.854184 24268 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bece0a1a4fa4017ad69b968101bb3d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:52 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:45432
I20260929 16:02:52.859491 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23937
I20260929 16:02:52.866863 24378 tablet_replica.cc:330] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: stopping tablet replica
I20260929 16:02:52.867381 24378 raft_consensus.cc:2229] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:52.868026 24377 tablet_replica.cc:330] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: stopping tablet replica
I20260929 16:02:52.868799 24377 raft_consensus.cc:2229] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:52.869485 24377 raft_consensus.cc:2258] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:52.870048 24378 raft_consensus.cc:2258] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:52.873835 24378 ts_tablet_manager.cc:1905] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:52.874565 24376 tablet_replica.cc:330] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: stopping tablet replica
I20260929 16:02:52.874997 24377 ts_tablet_manager.cc:1905] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:52.875536 24376 raft_consensus.cc:2229] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:52.883309 24377 ts_tablet_manager.cc:1918] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:52.883559 24377 log.cc:1196] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/6bece0a1a4fa4017ad69b968101bb3d4
I20260929 16:02:52.884249 24377 ts_tablet_manager.cc:1939] T 6bece0a1a4fa4017ad69b968101bb3d4 P 1df911250338423da7e33a786bba54c4: Deleting consensus metadata
I20260929 16:02:52.886335 23860 catalog_manager.cc:4841] TS 1df911250338423da7e33a786bba54c4 (127.5.117.1:46475): tablet 6bece0a1a4fa4017ad69b968101bb3d4 (table test-table [id=26c901f99e7f444596a4471762a6ac4e]) successfully deleted
I20260929 16:02:52.886548 24378 ts_tablet_manager.cc:1918] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:52.886700 24378 log.cc:1196] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/6bece0a1a4fa4017ad69b968101bb3d4
I20260929 16:02:52.887840 24376 raft_consensus.cc:2258] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:52.888935 24378 ts_tablet_manager.cc:1939] T 6bece0a1a4fa4017ad69b968101bb3d4 P bde81d48234a4e268049a5934fb286b1: Deleting consensus metadata
I20260929 16:02:52.890713 23859 catalog_manager.cc:4841] TS bde81d48234a4e268049a5934fb286b1 (127.5.117.3:36213): tablet 6bece0a1a4fa4017ad69b968101bb3d4 (table test-table [id=26c901f99e7f444596a4471762a6ac4e]) successfully deleted
I20260929 16:02:52.892890 24376 ts_tablet_manager.cc:1905] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:02:52 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:02:52.901626 24376 ts_tablet_manager.cc:1918] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:52.901889 24376 log.cc:1196] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/6bece0a1a4fa4017ad69b968101bb3d4
I20260929 16:02:52.902560 24376 ts_tablet_manager.cc:1939] T 6bece0a1a4fa4017ad69b968101bb3d4 P bbd72bbbbdec4b739e3fc1eff2ea020e: Deleting consensus metadata
I20260929 16:02:52.904453 23861 catalog_manager.cc:4841] TS bbd72bbbbdec4b739e3fc1eff2ea020e (127.5.117.2:36821): tablet 6bece0a1a4fa4017ad69b968101bb3d4 (table test-table [id=26c901f99e7f444596a4471762a6ac4e]) successfully deleted
I20260929 16:02:52.906231 24348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.861195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.5.117.1:46475)
0929 16:02:52.861433 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.861443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.878427 (+ 16984us) server_negotiation.cc:407] Connection header received
0929 16:02:52.878889 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.878895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.879011 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.879099 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:52.886995 (+ 7896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.888234 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.890262 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:52.890887 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:52.891007 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:52.898354 (+ 7347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:52.898369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:52.898391 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:52.898433 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:52.902731 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.903639 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.903649 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.903653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.903825 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:52.904428 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:52.904434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:52.904437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:52.905005 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:52.905351 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.905545 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.905823 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:02:52.907150 24060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:53.005606 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23937
I20260929 16:02:53.034447 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24071
Sep 29 16:02:53 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:02:53.052003 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:53.035429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.5.117.2:36821)
0929 16:02:53.035700 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:53.035708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:53.035741 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:02:53.035840 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:53.035852 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:53.035935 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:53.036106 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:53.037578 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:53.038533 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:53.040018 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:53.040542 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:53.040901 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:53.045624 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:53.045641 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:53.045664 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:53.045702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:53.049054 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:53.049765 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:53.049773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:53.049775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:53.049925 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:53.050319 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:53.050326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:53.050329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:53.050728 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:53.051071 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:53.051352 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:02:53.051582 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 16:02:53.053105 24194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:53.179626 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24071
I20260929 16:02:53.202963 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24205
Sep 29 16:02:53 dist-test-slave-b69v krb5kdc[23830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:02:53.219134 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:53.203828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.5.117.3:36213)
0929 16:02:53.204080 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:53.204089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:53.204183 (+ 94us) server_negotiation.cc:407] Connection header received
0929 16:02:53.204432 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:53.204436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:53.204511 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:53.204629 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:53.206083 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:53.206876 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:53.208262 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:53.208652 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:53.208790 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:53.213459 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:53.213471 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:53.213487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:53.213527 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:53.216306 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:53.217011 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:53.217019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:53.217021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:53.217154 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:53.217547 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:53.217557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:53.217559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:53.217902 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:53.218244 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:53.218538 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:02:53.218778 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:02:53.219834 24328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:53.315892 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24205
I20260929 16:02:53.336369 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 23843
I20260929 16:02:53.337639 23905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:53.466439 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 23843
2026-09-29T16:02:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4836 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:53Z Disabled control of system clock
I20260929 16:02:53.562016 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37083
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:53.938727 24387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:53.939044 24387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:53.939181 24387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:53.946672 24387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:53.946781 24387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:53.946870 24387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:53.946975 24387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:53.947046 24387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:53.956759 24387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:37083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:53.958346 24387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:53.960565 24387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:53.973779 24395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:53.973935 24392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:53.973843 24393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:53.974915 24387 server_base.cc:1034] running on GCE node
I20260929 16:02:53.975940 24387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:53.977689 24387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:53.978966 24387 hybrid_clock.cc:648] HybridClock initialized: now 1790697773978927 us; error 69 us; skew 500 ppm
I20260929 16:02:53.979501 24387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:53.993208 24387 webserver.cc:460] Webserver started at http://127.5.117.62:44423/ using document root <none> and password file <none>
I20260929 16:02:53.994132 24387 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:53.994284 24387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:53.994671 24387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:53.998196 24387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "d0150793a1584f1d9a6c7d0e66e2c738"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-b69v"
server_key: "d4e3f72e0537a2757cf0f1c3058eae36"
server_key_iv: "514b6525dfe3e1e55a27e60c1834b783"
server_key_version: "encryptionkey@0"
I20260929 16:02:53.999125 24387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "d0150793a1584f1d9a6c7d0e66e2c738"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-b69v"
server_key: "d4e3f72e0537a2757cf0f1c3058eae36"
server_key_iv: "514b6525dfe3e1e55a27e60c1834b783"
server_key_version: "encryptionkey@0"
I20260929 16:02:54.005046 24387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:02:54.008975 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:54.011023 24387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:02:54.011260 24387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "d0150793a1584f1d9a6c7d0e66e2c738"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-b69v"
server_key: "d4e3f72e0537a2757cf0f1c3058eae36"
server_key_iv: "514b6525dfe3e1e55a27e60c1834b783"
server_key_version: "encryptionkey@0"
I20260929 16:02:54.011528 24387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:54.032742 24387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:54.033937 24387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:54.034477 24387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:54.059952 24387 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:37083
I20260929 16:02:54.060003 24452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:37083 every 8 connection(s)
I20260929 16:02:54.061856 24387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:54.066717 24453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:54.067726 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24387
I20260929 16:02:54.067960 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:02:54.068548 5588 external_mini_cluster.cc:1428] Setting key fec9dd042f1d885f56dadbe92fa4841c
I20260929 16:02:54.082906 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: Bootstrap starting.
I20260929 16:02:54.086567 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:54.071283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.5.117.62:37083)
0929 16:02:54.072780 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:54.072792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:54.072839 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:02:54.074705 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:54.074731 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:54.075594 (+ 863us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:54.076130 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:54.076891 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.078322 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.080390 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.081140 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.081287 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:54.082925 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:54.082933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:54.082950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:54.083676 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:54.083767 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:54.083791 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:54.083896 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:54.084195 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 16:02:54.084608 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":756,"threads_started":1}
I20260929 16:02:54.087330 24453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:54.088770 24453 log.cc:824] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:54.092208 24453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: No bootstrap required, opened a new log
I20260929 16:02:54.100558 24453 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } }
I20260929 16:02:54.100935 24453 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:54.101096 24453 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0150793a1584f1d9a6c7d0e66e2c738, State: Initialized, Role: FOLLOWER
I20260929 16:02:54.101812 24453 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } }
I20260929 16:02:54.102053 24453 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:54.102159 24453 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:54.102288 24453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:54.104629 24453 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } }
I20260929 16:02:54.105122 24453 leader_election.cc:302] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0150793a1584f1d9a6c7d0e66e2c738; no voters:
I20260929 16:02:54.105674 24453 leader_election.cc:288] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:54.105868 24458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:54.107138 24458 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [term 1 LEADER]: Becoming Leader. State: Replica: d0150793a1584f1d9a6c7d0e66e2c738, State: Running, Role: LEADER
I20260929 16:02:54.107661 24458 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } }
I20260929 16:02:54.108227 24453 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:54.110702 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } } }
I20260929 16:02:54.110865 24460 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0150793a1584f1d9a6c7d0e66e2c738. Latest consensus state: current_term: 1 leader_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0150793a1584f1d9a6c7d0e66e2c738" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 37083 } } }
I20260929 16:02:54.111160 24460 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:54.111163 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:54.113510 24465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:54.118721 24465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:54.128655 24465 catalog_manager.cc:1347] Generated new cluster ID: e961ed5911d541f5a38966cb87a6c396
I20260929 16:02:54.128768 24465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:54.164201 24465 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:54.166299 24465 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:54.176488 24465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: Generated new TSK 0
I20260929 16:02:54.177433 24465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:54.192895 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--builtin_ntp_servers=127.5.117.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:54.591382 24477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:54.591710 24477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:54.591799 24477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:54.599313 24477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:54.599408 24477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:54.599608 24477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:54.608892 24477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:54.610436 24477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:54.612555 24477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:54.627945 24485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:54.627974 24482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:54.627856 24483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:54.628666 24477 server_base.cc:1034] running on GCE node
I20260929 16:02:54.629328 24477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:54.630239 24477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:54.631467 24477 hybrid_clock.cc:648] HybridClock initialized: now 1790697774631437 us; error 43 us; skew 500 ppm
I20260929 16:02:54.631812 24477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:54.634194 24477 webserver.cc:460] Webserver started at http://127.5.117.1:45297/ using document root <none> and password file <none>
I20260929 16:02:54.634932 24477 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:54.635073 24477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:54.635421 24477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:54.638561 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "9ed92186d0bc47e4978a04745ba9078e"
format_stamp: "Formatted at 2026-09-29 16:02:54 on dist-test-slave-b69v"
server_key: "10475e713e41074cc64749748307e632"
server_key_iv: "bf38c3315fb1059fddbbe3c38fdc20e3"
server_key_version: "encryptionkey@0"
I20260929 16:02:54.639382 24477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "9ed92186d0bc47e4978a04745ba9078e"
format_stamp: "Formatted at 2026-09-29 16:02:54 on dist-test-slave-b69v"
server_key: "10475e713e41074cc64749748307e632"
server_key_iv: "bf38c3315fb1059fddbbe3c38fdc20e3"
server_key_version: "encryptionkey@0"
I20260929 16:02:54.644613 24477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:02:54.648175 24491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:54.649544 24477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:54.649672 24477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "9ed92186d0bc47e4978a04745ba9078e"
format_stamp: "Formatted at 2026-09-29 16:02:54 on dist-test-slave-b69v"
server_key: "10475e713e41074cc64749748307e632"
server_key_iv: "bf38c3315fb1059fddbbe3c38fdc20e3"
server_key_version: "encryptionkey@0"
I20260929 16:02:54.649852 24477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:54.666417 24477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:54.667227 24477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:54.667477 24477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:54.668548 24477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:54.670076 24477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:54.670164 24477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:54.670249 24477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:54.670305 24477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:54.709529 24477 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36769
I20260929 16:02:54.709579 24603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36769 every 8 connection(s)
I20260929 16:02:54.711119 24477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:54.720834 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24477
I20260929 16:02:54.721050 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:54.721570 5588 external_mini_cluster.cc:1428] Setting key 3a6d745b146b2d66ec6d635ea92dcc18
I20260929 16:02:54.724660 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--builtin_ntp_servers=127.5.117.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:54.725086 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:54.714098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:55877 (local address 127.5.117.62:37083)
0929 16:02:54.714671 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:54.714676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:54.715947 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 16:02:54.717971 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:54.717976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:54.718075 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:54.718213 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:54.719339 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.720239 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.722043 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.722517 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.722667 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:54.724141 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:54.724147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:54.724159 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:54.724179 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:54.724224 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:54.724237 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:54.724335 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:54.724579 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:02:54.724687 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260929 16:02:54.726101 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:54.714737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37083 (local address 127.5.117.1:55877)
0929 16:02:54.715773 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:54.715822 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:54.717684 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:54.718463 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:54.718472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:54.718556 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:54.719169 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:54.719187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.720415 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.720419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:54.721887 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:54.721900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.722757 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.722767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:54.723021 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:54.723797 (+ 776us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:54.723799 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:54.723841 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:54.723917 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:54.724351 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:54.724366 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:54.724378 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:02:54.724566 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:02:54.724960 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":329,"threads_started":1}
I20260929 16:02:54.727851 24604 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37083
I20260929 16:02:54.728230 24604 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:54.729094 24604 heartbeater.cc:511] Master 127.5.117.62:37083 requested a full tablet report, sending...
I20260929 16:02:54.731765 24418 ts_manager.cc:194] Registered new tserver with Master: 9ed92186d0bc47e4978a04745ba9078e (127.5.117.1:36769)
I20260929 16:02:54.734222 24418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.1:55877
W20260929 16:02:55.092630 24608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:55.092908 24608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:55.093055 24608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:55.100483 24608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:55.100575 24608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:55.100783 24608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:55.110134 24608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:55.111567 24608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:55.113615 24608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:55.128997 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:55.129004 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:55.128948 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:55.133666 24608 server_base.cc:1034] running on GCE node
I20260929 16:02:55.134298 24608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:55.135298 24608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:55.136534 24608 hybrid_clock.cc:648] HybridClock initialized: now 1790697775136494 us; error 34 us; skew 500 ppm
I20260929 16:02:55.136883 24608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:55.139297 24608 webserver.cc:460] Webserver started at http://127.5.117.2:38069/ using document root <none> and password file <none>
I20260929 16:02:55.140028 24608 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:55.140214 24608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:55.140578 24608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:55.143602 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "e23748d562f94c1993d1b8d5f9503ea4"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "f5d8faa502d2aa41464a3dfdab45b889"
server_key_iv: "4f1f3e6da98089a4dec708ef3093412f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.144510 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "e23748d562f94c1993d1b8d5f9503ea4"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "f5d8faa502d2aa41464a3dfdab45b889"
server_key_iv: "4f1f3e6da98089a4dec708ef3093412f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.149891 24608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 16:02:55.153530 24622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.154978 24608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:55.155205 24608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "e23748d562f94c1993d1b8d5f9503ea4"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "f5d8faa502d2aa41464a3dfdab45b889"
server_key_iv: "4f1f3e6da98089a4dec708ef3093412f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.155431 24608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:55.199045 24608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:55.200037 24608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:55.200382 24608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:55.201426 24608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:55.202967 24608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:55.203088 24608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.203229 24608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:55.203320 24608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.238104 24608 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:38843
I20260929 16:02:55.238148 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:38843 every 8 connection(s)
I20260929 16:02:55.239439 24608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:55.240579 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24608
I20260929 16:02:55.240783 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:55.241362 5588 external_mini_cluster.cc:1428] Setting key dff2d08f28f8806b6c6017d7816f92a3
I20260929 16:02:55.245045 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--builtin_ntp_servers=127.5.117.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:55.255524 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.242421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:56343 (local address 127.5.117.62:37083)
0929 16:02:55.242925 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.242930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.244437 (+ 1507us) server_negotiation.cc:407] Connection header received
0929 16:02:55.247300 (+ 2863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.247304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.247376 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.247484 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:55.249037 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.249773 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.252304 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.252830 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.252929 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.254718 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:55.254723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:55.254726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:55.254747 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:55.254793 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:55.254806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:55.254876 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.255142 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.255233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":258,"threads_started":1}
I20260929 16:02:55.257083 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.243110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37083 (local address 127.5.117.2:56343)
0929 16:02:55.244265 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:55.244321 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:55.247002 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:55.247751 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:55.247764 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:55.247894 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:55.248838 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.248867 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.249939 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.249948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.252169 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.252185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.253061 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.253070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.253441 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.254322 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:55.254327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:55.254385 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:55.254493 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:55.254905 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:55.254918 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:55.254929 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:02:55.255109 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 16:02:55.255537 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":372,"threads_started":1}
I20260929 16:02:55.258607 24735 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37083
I20260929 16:02:55.258967 24735 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:55.259994 24735 heartbeater.cc:511] Master 127.5.117.62:37083 requested a full tablet report, sending...
I20260929 16:02:55.261767 24418 ts_manager.cc:194] Registered new tserver with Master: e23748d562f94c1993d1b8d5f9503ea4 (127.5.117.2:38843)
I20260929 16:02:55.262658 24418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.2:56343
W20260929 16:02:55.620414 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:55.620728 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:55.620885 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:55.628178 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:55.628274 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:55.628454 24739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:02:55.637430 24739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:37083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:55.638934 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:55.641048 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:55.654989 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:55.654953 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:55.654914 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:55.655807 24739 server_base.cc:1034] running on GCE node
I20260929 16:02:55.656539 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:55.657459 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:55.658680 24739 hybrid_clock.cc:648] HybridClock initialized: now 1790697775658640 us; error 54 us; skew 500 ppm
I20260929 16:02:55.659020 24739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:55.661376 24739 webserver.cc:460] Webserver started at http://127.5.117.3:43893/ using document root <none> and password file <none>
I20260929 16:02:55.662073 24739 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:55.662190 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:55.662539 24739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:55.665740 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "c79854572f8a4139bc1073bdf08a9c43"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "cff9f90a42958a7b4d6cd3f4585d7c99"
server_key_iv: "25ee30549a135b765f54a198faecf44f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.666633 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "c79854572f8a4139bc1073bdf08a9c43"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "cff9f90a42958a7b4d6cd3f4585d7c99"
server_key_iv: "25ee30549a135b765f54a198faecf44f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.671840 24739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:02:55.675364 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.676688 24739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:55.676877 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "c79854572f8a4139bc1073bdf08a9c43"
format_stamp: "Formatted at 2026-09-29 16:02:55 on dist-test-slave-b69v"
server_key: "cff9f90a42958a7b4d6cd3f4585d7c99"
server_key_iv: "25ee30549a135b765f54a198faecf44f"
server_key_version: "encryptionkey@0"
I20260929 16:02:55.677107 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:55.697619 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:55.698628 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:55.698962 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:55.700229 24739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:55.702085 24739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:55.702224 24739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.702358 24739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:55.702455 24739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:55.737989 24604 heartbeater.cc:503] Master 127.5.117.62:37083 was elected leader, sending a full tablet report...
I20260929 16:02:55.745391 24739 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:35003
I20260929 16:02:55.745432 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:35003 every 8 connection(s)
I20260929 16:02:55.746860 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:55.752257 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24739
I20260929 16:02:55.752517 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:55.753134 5588 external_mini_cluster.cc:1428] Setting key e5d3d32068bfa0516746f9de727756b3
I20260929 16:02:55.761945 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.749979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:49783 (local address 127.5.117.62:37083)
0929 16:02:55.750204 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.750209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.751777 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 16:02:55.754811 (+ 3034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.754815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.754887 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.754973 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:55.756401 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.757310 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.758919 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.759396 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.759538 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.761067 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:55.761072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:55.761075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:55.761094 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:55.761132 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:55.761145 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:55.761246 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.761487 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.761603 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:02:55.762934 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.750565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:37083 (local address 127.5.117.3:49783)
0929 16:02:55.751598 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:55.751645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:55.754411 (+ 2766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:55.755239 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:55.755247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:55.755335 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:55.756237 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.756265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.757477 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.757482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.758765 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.758778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.759635 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.759644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.759888 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.760715 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:55.760718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:55.760758 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:55.760855 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:55.761244 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:55.761256 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:55.761266 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:02:55.761447 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:02:55.761840 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260929 16:02:55.764273 24866 heartbeater.cc:347] Connected to a master server at 127.5.117.62:37083
I20260929 16:02:55.764549 24866 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:55.765375 24866 heartbeater.cc:511] Master 127.5.117.62:37083 requested a full tablet report, sending...
I20260929 16:02:55.766860 24418 ts_manager.cc:194] Registered new tserver with Master: c79854572f8a4139bc1073bdf08a9c43 (127.5.117.3:35003)
I20260929 16:02:55.767789 24418 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.3:49783
I20260929 16:02:55.770013 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:55.781378 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.775305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.5.117.62:37083)
0929 16:02:55.775480 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.775484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.775935 (+ 451us) server_negotiation.cc:407] Connection header received
0929 16:02:55.776110 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.776115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.776189 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.776271 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:55.776737 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.777471 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.778856 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.779281 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.779375 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.780601 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:55.780606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:55.780609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:55.780627 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:55.780670 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:55.780685 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:55.780789 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.780980 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.781067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:02:55.785352 24417 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:55.792901 24417 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:55.822702 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.805420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:38843 (local address 127.0.0.1:42014)
0929 16:02:55.806880 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:55.806933 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:55.807150 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:55.810048 (+ 2898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:55.810058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:55.810090 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:55.810698 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.810715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.813385 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.813404 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.816688 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.816711 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.819890 (+ 3179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.819898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.821756 (+ 1858us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.821831 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 16:02:55.821993 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 16:02:55.822243 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":1147,"thread_start_us":1010,"threads_started":1}
I20260929 16:02:55.826040 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.805596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.5.117.2:38843)
0929 16:02:55.809091 (+ 3495us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.809102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.809136 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:02:55.809244 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.809251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.809531 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.809759 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:55.810953 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.813148 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.817038 (+ 3890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.819038 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.820718 (+ 1680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.821514 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.823549 (+ 2035us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.825695 (+ 2146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3330,"thread_start_us":3220,"threads_started":1}
I20260929 16:02:55.830407 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.805606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:35003 (local address 127.0.0.1:56714)
0929 16:02:55.809305 (+ 3699us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:55.809325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:55.809533 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:55.812127 (+ 2594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:55.812135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:55.812171 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:55.812742 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.812757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.816981 (+ 4224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.816989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.825324 (+ 8335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.825344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.829463 (+ 4119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.829473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.829708 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.829756 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:02:55.829846 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:02:55.830029 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3576,"thread_start_us":3437,"threads_started":1}
I20260929 16:02:55.831822 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.809466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36769 (local address 127.0.0.1:52174)
0929 16:02:55.820002 (+ 10536us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:55.820022 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:55.821839 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:55.822950 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:55.822957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:55.822989 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:55.823427 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.823442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.826132 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.826139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.828918 (+ 2779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:55.828933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.831057 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.831065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:55.831264 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.831297 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:02:55.831401 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:02:55.831520 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10411,"thread_start_us":10243,"threads_started":1}
I20260929 16:02:55.831791 24670 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 (DEFAULT_TABLE table=test-table [id=5bd02898d5af4952ab9cebf309df1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:55.832988 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.809940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.5.117.1:36769)
0929 16:02:55.819678 (+ 9738us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.819690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.820251 (+ 561us) server_negotiation.cc:407] Connection header received
0929 16:02:55.822116 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.822127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.822397 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.822613 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:55.823628 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.825941 (+ 2313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.829139 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.830534 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.830665 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.830929 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.832409 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.832592 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9533,"thread_start_us":9371,"threads_started":1}
I20260929 16:02:55.834682 24670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c16da90d34c24932a9239d3a0a1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:55.834904 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:55.806000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.5.117.3:35003)
0929 16:02:55.808831 (+ 2831us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:55.808844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:55.809415 (+ 571us) server_negotiation.cc:407] Connection header received
0929 16:02:55.809773 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:55.809785 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:55.810057 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:55.811676 (+ 1619us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:55.813019 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.816742 (+ 3723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.826904 (+ 10162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:55.828811 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:55.828979 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:55.829318 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:55.830963 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0929 16:02:55.832492 (+ 1529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2595,"thread_start_us":2389,"threads_started":1}
I20260929 16:02:55.836804 24539 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 (DEFAULT_TABLE table=test-table [id=5bd02898d5af4952ab9cebf309df1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:55.838462 24539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c16da90d34c24932a9239d3a0a1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:55.839613 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 (DEFAULT_TABLE table=test-table [id=5bd02898d5af4952ab9cebf309df1014]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:55.842480 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b7c16da90d34c24932a9239d3a0a1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:55.857208 24884 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Bootstrap starting.
I20260929 16:02:55.863533 24886 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Bootstrap starting.
I20260929 16:02:55.864462 24885 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Bootstrap starting.
I20260929 16:02:55.865921 24884 tablet_bootstrap.cc:654] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:55.867422 24884 log.cc:824] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:55.867605 24886 tablet_bootstrap.cc:654] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:55.869035 24886 log.cc:824] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:55.869323 24885 tablet_bootstrap.cc:654] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:55.870854 24885 log.cc:824] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:55.876536 24885 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: No bootstrap required, opened a new log
I20260929 16:02:55.876981 24885 ts_tablet_manager.cc:1397] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260929 16:02:55.883153 24884 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: No bootstrap required, opened a new log
I20260929 16:02:55.883184 24886 tablet_bootstrap.cc:492] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: No bootstrap required, opened a new log
I20260929 16:02:55.883481 24886 ts_tablet_manager.cc:1397] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260929 16:02:55.883539 24884 ts_tablet_manager.cc:1397] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.015s
I20260929 16:02:55.894114 24885 raft_consensus.cc:348] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.894919 24885 raft_consensus.cc:374] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:55.895107 24885 raft_consensus.cc:729] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed92186d0bc47e4978a04745ba9078e, State: Initialized, Role: FOLLOWER
I20260929 16:02:55.895390 24884 raft_consensus.cc:348] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.895323 24886 raft_consensus.cc:348] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.895797 24886 raft_consensus.cc:374] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:55.895831 24884 raft_consensus.cc:374] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:55.895984 24886 raft_consensus.cc:729] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79854572f8a4139bc1073bdf08a9c43, State: Initialized, Role: FOLLOWER
I20260929 16:02:55.896018 24884 raft_consensus.cc:729] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23748d562f94c1993d1b8d5f9503ea4, State: Initialized, Role: FOLLOWER
I20260929 16:02:55.896200 24885 consensus_queue.cc:260] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.896728 24884 consensus_queue.cc:260] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.896735 24886 consensus_queue.cc:260] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:55.905055 24884 ts_tablet_manager.cc:1428] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Time spent starting tablet: real 0.021s user 0.018s sys 0.001s
I20260929 16:02:55.905061 24886 ts_tablet_manager.cc:1428] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260929 16:02:55.905067 24885 ts_tablet_manager.cc:1428] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Time spent starting tablet: real 0.028s user 0.023s sys 0.001s
I20260929 16:02:55.905517 24735 heartbeater.cc:503] Master 127.5.117.62:37083 was elected leader, sending a full tablet report...
I20260929 16:02:55.907315 24866 heartbeater.cc:503] Master 127.5.117.62:37083 was elected leader, sending a full tablet report...
W20260929 16:02:55.964198 24605 tablet.cc:2369] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:55.991820 24736 tablet.cc:2369] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:55.999109 24867 tablet.cc:2369] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:56.036412 24891 raft_consensus.cc:482] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:56.037007 24891 raft_consensus.cc:504] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:56.039255 24891 leader_election.cc:288] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e23748d562f94c1993d1b8d5f9503ea4 (127.5.117.2:38843), 9ed92186d0bc47e4978a04745ba9078e (127.5.117.1:36769)
I20260929 16:02:56.048310 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.039844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:56015 (local address 127.5.117.1:36769)
0929 16:02:56.040145 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.040155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.041257 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 16:02:56.041506 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.041513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.041613 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.041708 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:56.042837 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.043902 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.046111 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.047053 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.047197 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.047324 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.047796 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.047961 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:02:56.049047 24893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.039816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36769 (local address 127.5.117.3:56015)
0929 16:02:56.041144 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:56.041161 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:56.041347 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:56.041855 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:56.041862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:56.041899 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:56.042478 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:56.042493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.044099 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.044110 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:56.045889 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:56.045904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.047287 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.047296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:56.047541 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.047599 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:02:56.047729 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:02:56.047946 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":55,"thread_start_us":516,"threads_started":1}
I20260929 16:02:56.049562 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.040766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:38463 (local address 127.5.117.2:38843)
0929 16:02:56.041137 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.041144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.041173 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:02:56.041283 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.041292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.041408 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.041552 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:56.042363 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.043632 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.046927 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.048011 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.048171 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.048263 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.049146 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.049301 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 16:02:56.049947 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.039590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:38843 (local address 127.5.117.3:38463)
0929 16:02:56.040194 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:56.040222 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:56.040397 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:56.041581 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:56.041587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:56.041622 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:56.041904 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:56.041915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.043835 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.043843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:56.046702 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:56.046726 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.048369 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.048377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:56.048591 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.048625 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:02:56.048716 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:02:56.048873 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":101}
I20260929 16:02:56.051355 24559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c16da90d34c24932a9239d3a0a1c1" candidate_uuid: "c79854572f8a4139bc1073bdf08a9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed92186d0bc47e4978a04745ba9078e" is_pre_election: true
I20260929 16:02:56.051388 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c16da90d34c24932a9239d3a0a1c1" candidate_uuid: "c79854572f8a4139bc1073bdf08a9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23748d562f94c1993d1b8d5f9503ea4" is_pre_election: true
I20260929 16:02:56.052021 24559 raft_consensus.cc:2454] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79854572f8a4139bc1073bdf08a9c43 in term 0.
I20260929 16:02:56.052098 24690 raft_consensus.cc:2454] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79854572f8a4139bc1073bdf08a9c43 in term 0.
I20260929 16:02:56.053082 24756 leader_election.cc:302] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed92186d0bc47e4978a04745ba9078e, c79854572f8a4139bc1073bdf08a9c43; no voters:
I20260929 16:02:56.053690 24891 raft_consensus.cc:2781] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:56.053915 24891 raft_consensus.cc:482] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:56.054035 24891 raft_consensus.cc:3037] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:56.058547 24891 raft_consensus.cc:504] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:56.060149 24559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c16da90d34c24932a9239d3a0a1c1" candidate_uuid: "c79854572f8a4139bc1073bdf08a9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed92186d0bc47e4978a04745ba9078e"
I20260929 16:02:56.060380 24559 raft_consensus.cc:3037] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:56.063004 24559 raft_consensus.cc:2454] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79854572f8a4139bc1073bdf08a9c43 in term 1.
I20260929 16:02:56.063259 24891 leader_election.cc:288] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [CANDIDATE]: Term 1 election: Requested vote from peers e23748d562f94c1993d1b8d5f9503ea4 (127.5.117.2:38843), 9ed92186d0bc47e4978a04745ba9078e (127.5.117.1:36769)
I20260929 16:02:56.063470 24756 leader_election.cc:302] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed92186d0bc47e4978a04745ba9078e, c79854572f8a4139bc1073bdf08a9c43; no voters:
I20260929 16:02:56.063759 24891 raft_consensus.cc:2781] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:56.064338 24891 raft_consensus.cc:686] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 1 LEADER]: Becoming Leader. State: Replica: c79854572f8a4139bc1073bdf08a9c43, State: Running, Role: LEADER
I20260929 16:02:56.064998 24891 consensus_queue.cc:237] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } }
I20260929 16:02:56.065579 24690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b7c16da90d34c24932a9239d3a0a1c1" candidate_uuid: "c79854572f8a4139bc1073bdf08a9c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23748d562f94c1993d1b8d5f9503ea4"
I20260929 16:02:56.065907 24690 raft_consensus.cc:3037] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:56.070528 24690 raft_consensus.cc:2454] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79854572f8a4139bc1073bdf08a9c43 in term 1.
I20260929 16:02:56.071389 24416 catalog_manager.cc:5495] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 reported cstate change: term changed from 0 to 1, leader changed from <none> to c79854572f8a4139bc1073bdf08a9c43 (127.5.117.3). New cstate: current_term: 1 leader_uuid: "c79854572f8a4139bc1073bdf08a9c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c79854572f8a4139bc1073bdf08a9c43" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 } health_report { overall_health: UNKNOWN } } }
I20260929 16:02:56.121337 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.115730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.5.117.3:35003)
0929 16:02:56.115930 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.115937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.115970 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:02:56.116133 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.116137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.116252 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.116412 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:56.116875 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.117850 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.119182 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.119601 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.119696 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.120269 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:56.120836 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:56.120972 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.121103 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.121143 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:02:56.130357 24690 raft_consensus.cc:1261] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 1 FOLLOWER]: Refusing update from remote peer c79854572f8a4139bc1073bdf08a9c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:56.130357 24559 raft_consensus.cc:1261] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 1 FOLLOWER]: Refusing update from remote peer c79854572f8a4139bc1073bdf08a9c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:56.131390 24894 consensus_queue.cc:1035] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed92186d0bc47e4978a04745ba9078e" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:56.131767 24891 consensus_queue.cc:1035] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e23748d562f94c1993d1b8d5f9503ea4" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:56.145016 24899 mvcc.cc:205] Tried to move back new op lower bound from 7334698091018977280 to 7334698090767228928. Current Snapshot: MvccSnapshot[applied={T|T < 7334698091018977280}]
I20260929 16:02:56.159106 24416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:56.159441 24416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:56.163448 24416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0150793a1584f1d9a6c7d0e66e2c738: Sending DeleteTablet for 3 replicas of tablet 1b7c16da90d34c24932a9239d3a0a1c1
I20260929 16:02:56.164587 24670 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:56 UTC) from {username='slave'} at 127.0.0.1:42014
I20260929 16:02:56.164922 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:56 UTC) from {username='slave'} at 127.0.0.1:56714
I20260929 16:02:56.165132 24539 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b7c16da90d34c24932a9239d3a0a1c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:56 UTC) from {username='slave'} at 127.0.0.1:52174
I20260929 16:02:56.166250 24906 tablet_replica.cc:330] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: stopping tablet replica
I20260929 16:02:56.166499 24908 tablet_replica.cc:330] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: stopping tablet replica
I20260929 16:02:56.166833 24906 raft_consensus.cc:2229] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:56.166908 24908 raft_consensus.cc:2229] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:56.167435 24908 raft_consensus.cc:2258] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:56.167435 24906 raft_consensus.cc:2258] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:56.168308 24907 tablet_replica.cc:330] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: stopping tablet replica
I20260929 16:02:56.168680 24907 raft_consensus.cc:2229] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:56.169317 24907 raft_consensus.cc:2258] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:56.169368 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24477
I20260929 16:02:56.170359 24906 ts_tablet_manager.cc:1905] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:56.171212 24908 ts_tablet_manager.cc:1905] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:56.178912 24908 ts_tablet_manager.cc:1918] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:56.179103 24908 log.cc:1196] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/1b7c16da90d34c24932a9239d3a0a1c1
I20260929 16:02:56.179684 24906 ts_tablet_manager.cc:1918] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:56.179720 24908 ts_tablet_manager.cc:1939] T 1b7c16da90d34c24932a9239d3a0a1c1 P 9ed92186d0bc47e4978a04745ba9078e: Deleting consensus metadata
I20260929 16:02:56.180033 24906 log.cc:1196] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/1b7c16da90d34c24932a9239d3a0a1c1
I20260929 16:02:56.180882 24906 ts_tablet_manager.cc:1939] T 1b7c16da90d34c24932a9239d3a0a1c1 P e23748d562f94c1993d1b8d5f9503ea4: Deleting consensus metadata
I20260929 16:02:56.181829 24404 catalog_manager.cc:4841] TS 9ed92186d0bc47e4978a04745ba9078e (127.5.117.1:36769): tablet 1b7c16da90d34c24932a9239d3a0a1c1 (table test-table [id=5bd02898d5af4952ab9cebf309df1014]) successfully deleted
I20260929 16:02:56.182934 24405 catalog_manager.cc:4841] TS e23748d562f94c1993d1b8d5f9503ea4 (127.5.117.2:38843): tablet 1b7c16da90d34c24932a9239d3a0a1c1 (table test-table [id=5bd02898d5af4952ab9cebf309df1014]) successfully deleted
I20260929 16:02:56.183511 24907 ts_tablet_manager.cc:1905] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:56.190285 24907 ts_tablet_manager.cc:1918] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:56.190491 24907 log.cc:1196] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/1b7c16da90d34c24932a9239d3a0a1c1
I20260929 16:02:56.191025 24907 ts_tablet_manager.cc:1939] T 1b7c16da90d34c24932a9239d3a0a1c1 P c79854572f8a4139bc1073bdf08a9c43: Deleting consensus metadata
I20260929 16:02:56.191159 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.172089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.5.117.1:36769)
0929 16:02:56.172275 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.172285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.182753 (+ 10468us) server_negotiation.cc:407] Connection header received
0929 16:02:56.183063 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.183071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.183184 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.183257 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:56.183876 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.185082 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.187041 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.187687 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.187824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.189254 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:56.189263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:56.189281 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:56.190063 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:56.190136 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:56.190163 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:56.190285 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.190524 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.190745 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:02:56.191910 24599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:56.193043 24404 catalog_manager.cc:4841] TS c79854572f8a4139bc1073bdf08a9c43 (127.5.117.3:35003): tablet 1b7c16da90d34c24932a9239d3a0a1c1 (table test-table [id=5bd02898d5af4952ab9cebf309df1014]) successfully deleted
I20260929 16:02:56.300966 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24477
I20260929 16:02:56.323735 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24608
I20260929 16:02:56.331477 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.324834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.5.117.2:38843)
0929 16:02:56.325050 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.325056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.325078 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:02:56.325160 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.325166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.325289 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.325415 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:56.325883 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.326730 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.328120 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.328543 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.328675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.329850 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:56.329861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:56.329877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:56.330578 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:56.330629 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:56.330650 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:56.330777 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.330931 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.331066 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:02:56.332341 24730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:56.430989 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24608
I20260929 16:02:56.458941 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24739
I20260929 16:02:56.467041 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.459844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.5.117.3:35003)
0929 16:02:56.460177 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.460183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.460205 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:02:56.460406 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.460421 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.460512 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.460632 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:56.461094 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.461981 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.463334 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.463816 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.463963 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.465320 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:56.465333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:56.465351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:56.466102 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:56.466159 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:56.466183 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:56.466315 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.466461 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.466590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 16:02:56.468034 24861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:56.599442 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24739
I20260929 16:02:56.623103 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24387
I20260929 16:02:56.624423 24448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:56.746488 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24387
2026-09-29T16:02:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3279 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:56 dist-test-slave-b69v krb5kdc[24916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:56 dist-test-slave-b69v krb5kdc[24916](info): set up 2 sockets
Sep 29 16:02:56 dist-test-slave-b69v krb5kdc[24916](info): commencing operation
krb5kdc: starting...
W20260929 16:02:57.873059 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:02:58 dist-test-slave-b69v krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:02:58.197722 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35547
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:58.612967 24929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:58.613315 24929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:58.613504 24929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:58.621079 24929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:58.621193 24929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:58.621294 24929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:58.621407 24929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:58.621490 24929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:58.631996 24929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:58.633821 24929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:58.635418 24929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:58.648756 24935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:58.648711 24937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:58.648910 24934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:58.649709 24929 server_base.cc:1034] running on GCE node
I20260929 16:02:58.650483 24929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:58.651973 24929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:58.653175 24929 hybrid_clock.cc:648] HybridClock initialized: now 1790697778653135 us; error 33 us; skew 500 ppm
I20260929 16:02:58.653581 24929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:58 dist-test-slave-b69v krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:58.656939 24929 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:02:58.658547 24929 webserver.cc:460] Webserver started at http://127.5.117.62:39317/ using document root <none> and password file <none>
I20260929 16:02:58.659272 24929 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:58.659387 24929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:58.659673 24929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:58.663082 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "d2b644931304411587d860ad2bef1142"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-b69v"
server_key: "555477a1954a81525d0e82521d00b939"
server_key_iv: "c3c8bfc2677e4ef8d6da6c7b7f4f1586"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.663895 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "d2b644931304411587d860ad2bef1142"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-b69v"
server_key: "555477a1954a81525d0e82521d00b939"
server_key_iv: "c3c8bfc2677e4ef8d6da6c7b7f4f1586"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.669606 24929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:02:58.673346 24944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:58.674749 24929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:58.674934 24929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "d2b644931304411587d860ad2bef1142"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-b69v"
server_key: "555477a1954a81525d0e82521d00b939"
server_key_iv: "c3c8bfc2677e4ef8d6da6c7b7f4f1586"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.675129 24929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:58.694941 24929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:58.700220 24929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:58.700513 24929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:58.723493 24929 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:35547
I20260929 16:02:58.723637 24995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:35547 every 8 connection(s)
I20260929 16:02:58.725250 24929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:58.729984 24996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:58.730788 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 24929
I20260929 16:02:58.731019 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:02:58.731701 5588 external_mini_cluster.cc:1428] Setting key 7f7e5d8bbf60ab787724a878372a9313
I20260929 16:02:58.747938 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: Bootstrap starting.
Sep 29 16:02:58 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:58.751873 24996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:58.753358 24996 log.cc:824] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:58.757349 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: No bootstrap required, opened a new log
I20260929 16:02:58.759984 24999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:58.734046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.5.117.62:35547)
0929 16:02:58.735088 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:58.735107 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:58.735156 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:02:58.737273 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:58.737300 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:58.737821 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:58.738282 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:58.740637 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:58.742154 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:58.744269 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:58.745080 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:58.745234 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:58.750780 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:58.750789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:58.750809 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:58.750860 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:58.755113 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:58.755922 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:58.755929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:58.755937 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:58.756189 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:58.756645 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:58.756649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:58.756653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:58.757224 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:58.757563 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:58.758086 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 16:02:58.758593 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":485,"threads_started":1}
I20260929 16:02:58.767995 24996 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } }
I20260929 16:02:58.768576 24996 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:58.768703 24996 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b644931304411587d860ad2bef1142, State: Initialized, Role: FOLLOWER
I20260929 16:02:58.769341 24996 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } }
I20260929 16:02:58.769637 24996 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:58.769799 24996 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:58.769991 24996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:58.772823 24996 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } }
I20260929 16:02:58.773559 24996 leader_election.cc:302] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b644931304411587d860ad2bef1142; no voters:
I20260929 16:02:58.774164 24996 leader_election.cc:288] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:58.774322 25001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:58.775508 25001 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [term 1 LEADER]: Becoming Leader. State: Replica: d2b644931304411587d860ad2bef1142, State: Running, Role: LEADER
I20260929 16:02:58.776356 25001 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } }
I20260929 16:02:58.777099 24996 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:58.779946 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b644931304411587d860ad2bef1142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } } }
I20260929 16:02:58.780476 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:58.780333 25003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b644931304411587d860ad2bef1142. Latest consensus state: current_term: 1 leader_uuid: "d2b644931304411587d860ad2bef1142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b644931304411587d860ad2bef1142" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35547 } } }
I20260929 16:02:58.780635 25003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:58.783030 25009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:58.788499 25009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:58.800551 25009 catalog_manager.cc:1347] Generated new cluster ID: 271f3341c0a549eaa2080a65d51d8914
I20260929 16:02:58.800761 25009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:58.822752 25009 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:58.824630 25009 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:58.833508 25009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: Generated new TSK 0
I20260929 16:02:58.834445 25009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.117.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
Entry for principal kudu/127.5.117.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab.
I20260929 16:02:58.919937 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:59.300290 25022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:59.300648 25022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:59.300822 25022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:59.307907 25022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:59.308107 25022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:59.308311 25022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:02:59.318482 25022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.1.keytab
--principal=kudu/127.5.117.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:59.320170 25022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:59.321713 25022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:59.336565 25030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.336793 25027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.336763 25028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:59.337631 25022 server_base.cc:1034] running on GCE node
I20260929 16:02:59.338255 25022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:59.339248 25022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:59.340446 25022 hybrid_clock.cc:648] HybridClock initialized: now 1790697779340407 us; error 28 us; skew 500 ppm
I20260929 16:02:59.340828 25022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:59 dist-test-slave-b69v krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:59.350131 25022 init.cc:377] Logged in from keytab as kudu/127.5.117.1@KRBTEST.COM (short username kudu)
I20260929 16:02:59.351732 25022 webserver.cc:460] Webserver started at http://127.5.117.1:42493/ using document root <none> and password file <none>
I20260929 16:02:59.352648 25022 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:59.352792 25022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:59.353163 25022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:59.356634 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "61ebb8cb294b4b0bbe1ca847373af8ac"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "c74c67e15bfb629be27857a5ede1d8d5"
server_key_iv: "0d4d2cc99cab233f98a36c074fcf3966"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.357587 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "61ebb8cb294b4b0bbe1ca847373af8ac"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "c74c67e15bfb629be27857a5ede1d8d5"
server_key_iv: "0d4d2cc99cab233f98a36c074fcf3966"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.363865 25022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:02:59.368388 25037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:59.370400 25022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:02:59.370625 25022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "61ebb8cb294b4b0bbe1ca847373af8ac"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "c74c67e15bfb629be27857a5ede1d8d5"
server_key_iv: "0d4d2cc99cab233f98a36c074fcf3966"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.370891 25022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:59.386857 25022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:59.391805 25022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:59.392227 25022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:59.393435 25022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:59.395076 25022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:59.395185 25022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:59.395350 25022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:59.395430 25022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:59.435160 25022 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:35515
I20260929 16:02:59.435178 25149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:35515 every 8 connection(s)
I20260929 16:02:59.436760 25022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:59.437300 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25022
I20260929 16:02:59.437529 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:59.438140 5588 external_mini_cluster.cc:1428] Setting key ed664dcb71d148b1c8527d8fc7cbf2ff
Sep 29 16:02:59 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.1@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:02:59.462057 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:59.439975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:39059 (local address 127.5.117.62:35547)
0929 16:02:59.440574 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:59.440581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:59.441885 (+ 1304us) server_negotiation.cc:407] Connection header received
0929 16:02:59.445093 (+ 3208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:59.445098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:59.445165 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:59.445266 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:59.447531 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:59.448400 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:59.450891 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:59.451299 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:59.451401 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:59.456014 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:59.456019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:59.456023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:59.456101 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:59.458865 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:59.459575 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:59.459582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:59.459585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:59.459719 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:59.460229 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:59.460236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:59.460239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:59.460549 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:59.460715 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:59.461447 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 16:02:59.461684 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":276,"threads_started":1}
I20260929 16:02:59.463083 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:59.440731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35547 (local address 127.5.117.1:39059)
0929 16:02:59.441668 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:59.441738 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:59.444787 (+ 3049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:59.445574 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:59.445590 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:59.446436 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:59.447353 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:59.447382 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:59.448567 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:59.448575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:59.450746 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:59.450765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:59.451565 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:59.451575 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:59.451912 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:59.452765 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:59.452803 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:59.455770 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:59.459004 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:59.459012 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:59.459028 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:59.459456 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:59.459853 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:59.459857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:59.459861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:59.460114 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:59.460676 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:59.460684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:59.461057 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 16:02:59.461326 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 16:02:59.461883 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":345,"threads_started":1}
I20260929 16:02:59.464797 25150 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35547
I20260929 16:02:59.465152 25150 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:59.465963 25150 heartbeater.cc:511] Master 127.5.117.62:35547 requested a full tablet report, sending...
I20260929 16:02:59.468353 24961 ts_manager.cc:194] Registered new tserver with Master: 61ebb8cb294b4b0bbe1ca847373af8ac (127.5.117.1:35515)
I20260929 16:02:59.469957 24961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.1@KRBTEST.COM'} at 127.5.117.1:39059
WARNING: no policy specified for kudu/127.5.117.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
Entry for principal kudu/127.5.117.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab.
I20260929 16:02:59.511416 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:59.894994 25156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:59.895327 25156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:59.895495 25156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:59.903417 25156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:59.903563 25156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:59.903750 25156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:02:59.914685 25156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.2.keytab
--principal=kudu/127.5.117.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:02:59.916435 25156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:59.917985 25156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:59.932864 25161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.932957 25162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.932920 25164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:59.934026 25156 server_base.cc:1034] running on GCE node
I20260929 16:02:59.934676 25156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:59.935638 25156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:59.936834 25156 hybrid_clock.cc:648] HybridClock initialized: now 1790697779936772 us; error 49 us; skew 500 ppm
I20260929 16:02:59.937181 25156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:59 dist-test-slave-b69v krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:59.940543 25156 init.cc:377] Logged in from keytab as kudu/127.5.117.2@KRBTEST.COM (short username kudu)
I20260929 16:02:59.942118 25156 webserver.cc:460] Webserver started at http://127.5.117.2:39905/ using document root <none> and password file <none>
I20260929 16:02:59.942893 25156 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:59.943049 25156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:59.943395 25156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:59.946766 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "f1e583d03c7442b488f94d519db8990a"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "b7660c85876d82726ce1f5357a254e17"
server_key_iv: "39527e4cb4e5393dc7d96a56a9f61c0a"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.947642 25156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "f1e583d03c7442b488f94d519db8990a"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "b7660c85876d82726ce1f5357a254e17"
server_key_iv: "39527e4cb4e5393dc7d96a56a9f61c0a"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.953344 25156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 16:02:59.957147 25171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:59.958578 25156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:59.958760 25156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "f1e583d03c7442b488f94d519db8990a"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-b69v"
server_key: "b7660c85876d82726ce1f5357a254e17"
server_key_iv: "39527e4cb4e5393dc7d96a56a9f61c0a"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.958964 25156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:59.978253 25156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:59.982669 25156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:59.982987 25156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:59.984143 25156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:59.985592 25156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:59.985667 25156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:59.985844 25156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:59.985898 25156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:00.023623 25156 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:44973
I20260929 16:03:00.023660 25283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:44973 every 8 connection(s)
I20260929 16:03:00.025261 25156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:03:00.025789 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25156
I20260929 16:03:00.026072 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:03:00.026820 5588 external_mini_cluster.cc:1428] Setting key 9d4c26afad47a85846cbdf1f500f643d
Sep 29 16:03:00 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.2@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:00.053059 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.029044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:34815 (local address 127.5.117.62:35547)
0929 16:03:00.029660 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.029667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.030975 (+ 1308us) server_negotiation.cc:407] Connection header received
0929 16:03:00.034252 (+ 3277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.034257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.034331 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.034446 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:00.036885 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.037730 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.040414 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.040900 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.041013 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.045854 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:00.045862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:00.045871 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:00.045916 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:00.048899 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.049926 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.049930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.049932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.050066 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.050590 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.050594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.050596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.050899 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:00.051020 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.052013 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.052545 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":305,"threads_started":1}
I20260929 16:03:00.054369 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.029619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35547 (local address 127.5.117.2:34815)
0929 16:03:00.030740 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:00.030805 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:00.033895 (+ 3090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:00.034768 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:00.034782 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:00.035685 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:00.036649 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.036679 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.037898 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.037906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.040236 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.040257 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.041148 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.041157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.041508 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.042367 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:00.042413 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:00.045486 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:00.049054 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:00.049066 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:00.049089 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:00.049756 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:00.050189 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:00.050197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:00.050203 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:00.050458 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:00.051037 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:00.051047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:00.051556 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 16:03:00.051950 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 16:03:00.052665 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":383,"threads_started":1}
I20260929 16:03:00.056336 25284 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35547
I20260929 16:03:00.056785 25284 heartbeater.cc:464] Registering TS with master...
I20260929 16:03:00.057879 25284 heartbeater.cc:511] Master 127.5.117.62:35547 requested a full tablet report, sending...
I20260929 16:03:00.061132 24961 ts_manager.cc:194] Registered new tserver with Master: f1e583d03c7442b488f94d519db8990a (127.5.117.2:44973)
I20260929 16:03:00.062618 24961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.2@KRBTEST.COM'} at 127.5.117.2:34815
WARNING: no policy specified for kudu/127.5.117.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
Entry for principal kudu/127.5.117.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab.
I20260929 16:03:00.110001 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:00.473362 25150 heartbeater.cc:503] Master 127.5.117.62:35547 was elected leader, sending a full tablet report...
W20260929 16:03:00.494046 25290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:00.494362 25290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:00.494475 25290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:00.501814 25290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:00.501930 25290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:00.502045 25290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:03:00.512555 25290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.3.keytab
--principal=kudu/127.5.117.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:35547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:00.514165 25290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:00.515681 25290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:00.530351 25295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:00.530385 25296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:00.530448 25298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:00.531210 25290 server_base.cc:1034] running on GCE node
I20260929 16:03:00.531847 25290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:00.532989 25290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:00.534232 25290 hybrid_clock.cc:648] HybridClock initialized: now 1790697780534195 us; error 38 us; skew 500 ppm
I20260929 16:03:00.534576 25290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:00 dist-test-slave-b69v krb5kdc[24916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697780, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:00.538096 25290 init.cc:377] Logged in from keytab as kudu/127.5.117.3@KRBTEST.COM (short username kudu)
I20260929 16:03:00.539708 25290 webserver.cc:460] Webserver started at http://127.5.117.3:44085/ using document root <none> and password file <none>
I20260929 16:03:00.540541 25290 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:00.540648 25290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:00.540928 25290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:00.544301 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "56de784dae9747df9c0f905debfee605"
format_stamp: "Formatted at 2026-09-29 16:03:00 on dist-test-slave-b69v"
server_key: "9599bfca0e88a2ff1e59272d9a961b5e"
server_key_iv: "e26899255a66ed52ca91fa9c4bfb216c"
server_key_version: "encryptionkey@0"
I20260929 16:03:00.545099 25290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "56de784dae9747df9c0f905debfee605"
format_stamp: "Formatted at 2026-09-29 16:03:00 on dist-test-slave-b69v"
server_key: "9599bfca0e88a2ff1e59272d9a961b5e"
server_key_iv: "e26899255a66ed52ca91fa9c4bfb216c"
server_key_version: "encryptionkey@0"
I20260929 16:03:00.550536 25290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:00.554397 25305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:00.555976 25290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:03:00.556236 25290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "56de784dae9747df9c0f905debfee605"
format_stamp: "Formatted at 2026-09-29 16:03:00 on dist-test-slave-b69v"
server_key: "9599bfca0e88a2ff1e59272d9a961b5e"
server_key_iv: "e26899255a66ed52ca91fa9c4bfb216c"
server_key_version: "encryptionkey@0"
I20260929 16:03:00.556456 25290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:00.575208 25290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:00.580787 25290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:00.581111 25290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:00.582175 25290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:03:00.583624 25290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:03:00.583699 25290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:00.583876 25290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:03:00.583952 25290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:00.621763 25290 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43399
I20260929 16:03:00.621797 25417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43399 every 8 connection(s)
I20260929 16:03:00.623275 25290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:03:00.625252 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25290
I20260929 16:03:00.625470 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:03:00.626035 5588 external_mini_cluster.cc:1428] Setting key bfb395e024a288d534730d07b0bc3174
Sep 29 16:03:00 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697780, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.3@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:00.645952 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.626349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:37859 (local address 127.5.117.62:35547)
0929 16:03:00.626935 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.626942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.628195 (+ 1253us) server_negotiation.cc:407] Connection header received
0929 16:03:00.630459 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.630464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.630559 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.630664 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:00.632995 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.633845 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.635425 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.635847 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.635948 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.639657 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:00.639663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:00.639666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:00.639700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:00.642905 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.643614 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.643618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.643620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.643745 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.644329 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.644333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.644335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.644633 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:00.644769 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.645397 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.645580 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
I20260929 16:03:00.646914 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.626923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:35547 (local address 127.5.117.3:37859)
0929 16:03:00.627964 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:00.628024 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:00.630154 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:00.631209 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:00.631223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:00.632114 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:00.632796 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.632817 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.634003 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.634006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.635284 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.635296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.636092 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.636101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.636317 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.637015 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:00.637044 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:00.639410 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:00.643046 (+ 3636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:00.643053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:00.643064 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:00.643495 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:00.643837 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:00.643841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:00.643843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:00.644030 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:00.644748 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:00.644753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:00.645071 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 16:03:00.645307 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 16:03:00.645710 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":433,"threads_started":1}
I20260929 16:03:00.648495 25418 heartbeater.cc:347] Connected to a master server at 127.5.117.62:35547
I20260929 16:03:00.648885 25418 heartbeater.cc:464] Registering TS with master...
I20260929 16:03:00.649739 25418 heartbeater.cc:511] Master 127.5.117.62:35547 requested a full tablet report, sending...
I20260929 16:03:00.651414 24961 ts_manager.cc:194] Registered new tserver with Master: 56de784dae9747df9c0f905debfee605 (127.5.117.3:43399)
I20260929 16:03:00.652406 24961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.117.3@KRBTEST.COM'} at 127.5.117.3:37859
I20260929 16:03:00.654412 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:03:00.673749 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.659671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.5.117.62:35547)
0929 16:03:00.659900 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.659908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.660476 (+ 568us) server_negotiation.cc:407] Connection header received
0929 16:03:00.660712 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.660718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.660805 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.660922 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:00.662659 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.663449 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.664898 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.665348 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.665449 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.668129 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:00.668135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:00.668138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:00.668172 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:00.670907 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.671654 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.671658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.671661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.671802 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.672175 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.672178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.672180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.672606 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:00.672830 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.673226 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.673436 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:03:00.679282 24960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:03:00.687253 24960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:03:00.725703 25431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.702286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43399 (local address 127.0.0.1:50388)
0929 16:03:00.715688 (+ 13402us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:00.715720 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:00.715859 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:00.716997 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:00.717001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:00.717020 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:00.717585 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.717598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.719744 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.719750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.722218 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.722229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.724530 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.724537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.724747 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.724798 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:03:00.724942 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:03:00.725157 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13272,"mutex_wait_us":8,"thread_start_us":11856,"threads_started":1}
I20260929 16:03:00.726569 25432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.702229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:44973 (local address 127.0.0.1:52882)
0929 16:03:00.714664 (+ 12435us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:00.714742 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:00.715004 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:00.716698 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:00.716708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:00.716747 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:00.717336 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.717362 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.719744 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.719750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.722523 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.722538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.725726 (+ 3188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.725735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.725915 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.725945 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:03:00.726009 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:03:00.726120 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12053,"mutex_wait_us":57,"thread_start_us":13183,"threads_started":1}
I20260929 16:03:00.727460 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.703005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.5.117.3:43399)
0929 16:03:00.714162 (+ 11157us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.714175 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.715983 (+ 1808us) server_negotiation.cc:407] Connection header received
0929 16:03:00.716130 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.716139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.716433 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.716655 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:00.717811 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.719573 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.722377 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.723923 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.724107 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.724390 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.726617 (+ 2227us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.727059 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10904,"thread_start_us":10697,"threads_started":1}
I20260929 16:03:00.728152 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.702730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.5.117.2:44973)
0929 16:03:00.716119 (+ 13389us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.716128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.716154 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:03:00.716259 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.716265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.716448 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.717587 (+ 1139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:00.717832 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.719576 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.722698 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.725471 (+ 2773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.726795 (+ 1324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.726982 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.727060 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.727232 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13167,"thread_start_us":13035,"threads_started":1}
I20260929 16:03:00.729709 25436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.716209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:35515 (local address 127.0.0.1:40184)
0929 16:03:00.719816 (+ 3607us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:00.719839 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:00.719979 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:00.720853 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:00.720856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:00.720878 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:00.721262 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.721301 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.723208 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.723213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.725524 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:00.725554 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.728916 (+ 3362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.728924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:00.729106 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.729137 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:03:00.729245 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:03:00.729368 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3492,"thread_start_us":3379,"threads_started":1}
I20260929 16:03:00.729902 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.715802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.5.117.1:35515)
0929 16:03:00.719566 (+ 3764us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.719577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.720024 (+ 447us) server_negotiation.cc:407] Connection header received
0929 16:03:00.720167 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.720173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.720377 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.720572 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:00.721488 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.723042 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.725785 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.727358 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.727488 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.727716 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.729319 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.729530 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3564,"thread_start_us":3404,"threads_started":1}
I20260929 16:03:00.732249 25353 tablet_service.cc:1467] Processing CreateTablet for tablet 25f76502de1049249228677086fb9497 (DEFAULT_TABLE table=test-table [id=2ab2cc6469514067a10ab46410a62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:00.732256 25219 tablet_service.cc:1467] Processing CreateTablet for tablet 25f76502de1049249228677086fb9497 (DEFAULT_TABLE table=test-table [id=2ab2cc6469514067a10ab46410a62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:00.733355 25085 tablet_service.cc:1467] Processing CreateTablet for tablet 25f76502de1049249228677086fb9497 (DEFAULT_TABLE table=test-table [id=2ab2cc6469514067a10ab46410a62b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:00.734467 25353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f76502de1049249228677086fb9497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:00.734704 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f76502de1049249228677086fb9497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:00.734735 25085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f76502de1049249228677086fb9497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:00.754585 25439 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Bootstrap starting.
I20260929 16:03:00.755646 25438 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Bootstrap starting.
I20260929 16:03:00.756860 25437 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Bootstrap starting.
I20260929 16:03:00.759330 25439 tablet_bootstrap.cc:654] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:00.759636 25438 tablet_bootstrap.cc:654] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:00.760489 25437 tablet_bootstrap.cc:654] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:00.760874 25438 log.cc:824] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:00.760839 25439 log.cc:824] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:00.761950 25437 log.cc:824] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:00.766997 25438 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: No bootstrap required, opened a new log
I20260929 16:03:00.767151 25437 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: No bootstrap required, opened a new log
I20260929 16:03:00.767288 25439 tablet_bootstrap.cc:492] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: No bootstrap required, opened a new log
I20260929 16:03:00.767405 25438 ts_tablet_manager.cc:1397] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260929 16:03:00.767447 25437 ts_tablet_manager.cc:1397] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:03:00.767551 25439 ts_tablet_manager.cc:1397] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260929 16:03:00.775954 25438 raft_consensus.cc:348] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.776499 25438 raft_consensus.cc:374] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:00.776610 25438 raft_consensus.cc:729] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ebb8cb294b4b0bbe1ca847373af8ac, State: Initialized, Role: FOLLOWER
I20260929 16:03:00.777307 25438 consensus_queue.cc:260] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.778755 25439 raft_consensus.cc:348] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.778764 25437 raft_consensus.cc:348] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.779197 25439 raft_consensus.cc:374] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:00.779192 25437 raft_consensus.cc:374] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:00.779292 25437 raft_consensus.cc:729] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e583d03c7442b488f94d519db8990a, State: Initialized, Role: FOLLOWER
I20260929 16:03:00.779289 25439 raft_consensus.cc:729] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56de784dae9747df9c0f905debfee605, State: Initialized, Role: FOLLOWER
I20260929 16:03:00.779629 25438 ts_tablet_manager.cc:1428] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260929 16:03:00.779867 25439 consensus_queue.cc:260] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.780068 25437 consensus_queue.cc:260] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:00.781121 25418 heartbeater.cc:503] Master 127.5.117.62:35547 was elected leader, sending a full tablet report...
I20260929 16:03:00.781937 25439 ts_tablet_manager.cc:1428] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 16:03:00.783128 25284 heartbeater.cc:503] Master 127.5.117.62:35547 was elected leader, sending a full tablet report...
I20260929 16:03:00.783814 25437 ts_tablet_manager.cc:1428] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Time spent starting tablet: real 0.016s user 0.014s sys 0.004s
W20260929 16:03:00.875224 25419 tablet.cc:2369] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:03:00.940119 25151 tablet.cc:2369] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:01.002127 25445 raft_consensus.cc:482] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:03:01.002529 25445 raft_consensus.cc:504] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:01.004756 25445 leader_election.cc:288] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56de784dae9747df9c0f905debfee605 (127.5.117.3:43399), 61ebb8cb294b4b0bbe1ca847373af8ac (127.5.117.1:35515)
I20260929 16:03:01.012698 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.005334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:37203 (local address 127.5.117.3:43399)
0929 16:03:01.005604 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.005609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.006347 (+ 738us) server_negotiation.cc:407] Connection header received
0929 16:03:01.006654 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.006664 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.006769 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.006844 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:01.007666 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.008564 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.010614 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.011500 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.011670 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.011805 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.012224 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.012389 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 16:03:01.012781 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.006784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:50833 (local address 127.5.117.1:35515)
0929 16:03:01.007040 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.007047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.007174 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:03:01.007454 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.007458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.007529 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.007630 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:01.008232 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.008963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.010849 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.011698 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.011866 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.011992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.012357 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.012501 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:03:01.013047 25446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.005305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43399 (local address 127.5.117.2:37203)
0929 16:03:01.006171 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:01.006196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:01.006381 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:01.007092 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:01.007095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:01.007118 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:01.007497 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:01.007505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.008712 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.008720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:01.010422 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:01.010445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.011750 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.011756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:01.011952 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.011979 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:03:01.012185 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 16:03:01.012355 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":29,"spinlock_wait_cycles":5376,"thread_start_us":486,"threads_started":1}
I20260929 16:03:01.013155 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.006311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:35515 (local address 127.5.117.2:50833)
0929 16:03:01.007036 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:01.007055 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:01.007278 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:01.007657 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:01.007662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:01.007685 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:01.008026 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:01.008035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.009119 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.009125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:01.010666 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:01.010678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.011939 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.011945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:01.012183 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.012207 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:03:01.012331 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:03:01.012461 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":47,"thread_start_us":383,"threads_started":1}
I20260929 16:03:01.014232 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f76502de1049249228677086fb9497" candidate_uuid: "f1e583d03c7442b488f94d519db8990a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56de784dae9747df9c0f905debfee605" is_pre_election: true
I20260929 16:03:01.014300 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f76502de1049249228677086fb9497" candidate_uuid: "f1e583d03c7442b488f94d519db8990a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" is_pre_election: true
I20260929 16:03:01.014859 25105 raft_consensus.cc:2454] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e583d03c7442b488f94d519db8990a in term 0.
I20260929 16:03:01.014859 25373 raft_consensus.cc:2454] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1e583d03c7442b488f94d519db8990a in term 0.
I20260929 16:03:01.015599 25172 leader_election.cc:302] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61ebb8cb294b4b0bbe1ca847373af8ac, f1e583d03c7442b488f94d519db8990a; no voters:
I20260929 16:03:01.016041 25445 raft_consensus.cc:2781] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:03:01.016288 25445 raft_consensus.cc:482] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:03:01.016407 25445 raft_consensus.cc:3037] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:01.019656 25445 raft_consensus.cc:504] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
I20260929 16:03:01.020381 25445 leader_election.cc:288] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [CANDIDATE]: Term 1 election: Requested vote from peers 56de784dae9747df9c0f905debfee605 (127.5.117.3:43399), 61ebb8cb294b4b0bbe1ca847373af8ac (127.5.117.1:35515)
I20260929 16:03:01.021181 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f76502de1049249228677086fb9497" candidate_uuid: "f1e583d03c7442b488f94d519db8990a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56de784dae9747df9c0f905debfee605"
I20260929 16:03:01.021353 25105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f76502de1049249228677086fb9497" candidate_uuid: "f1e583d03c7442b488f94d519db8990a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac"
I20260929 16:03:01.021514 25373 raft_consensus.cc:3037] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:01.021616 25105 raft_consensus.cc:3037] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:01.024309 25373 raft_consensus.cc:2454] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e583d03c7442b488f94d519db8990a in term 1.
I20260929 16:03:01.024917 25172 leader_election.cc:302] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56de784dae9747df9c0f905debfee605, f1e583d03c7442b488f94d519db8990a; no voters:
I20260929 16:03:01.024957 25105 raft_consensus.cc:2454] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1e583d03c7442b488f94d519db8990a in term 1.
I20260929 16:03:01.025372 25445 raft_consensus.cc:2781] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:01.025949 25445 raft_consensus.cc:686] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 1 LEADER]: Becoming Leader. State: Replica: f1e583d03c7442b488f94d519db8990a, State: Running, Role: LEADER
I20260929 16:03:01.026880 25445 consensus_queue.cc:237] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } }
W20260929 16:03:01.028312 25285 tablet.cc:2369] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:01.032994 24961 catalog_manager.cc:5495] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a reported cstate change: term changed from 0 to 1, leader changed from <none> to f1e583d03c7442b488f94d519db8990a (127.5.117.2). New cstate: current_term: 1 leader_uuid: "f1e583d03c7442b488f94d519db8990a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1e583d03c7442b488f94d519db8990a" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 44973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 } health_report { overall_health: UNKNOWN } } }
I20260929 16:03:01.103735 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.098870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.5.117.2:44973)
0929 16:03:01.099067 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.099073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.099090 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:03:01.099204 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.099208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.099293 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.099436 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:01.099762 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.100537 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.101785 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.102166 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.102250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.102667 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:01.103223 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:01.103343 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.103426 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.103495 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:03:01.113023 25105 raft_consensus.cc:1261] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 1 FOLLOWER]: Refusing update from remote peer f1e583d03c7442b488f94d519db8990a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:01.113060 25373 raft_consensus.cc:1261] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 1 FOLLOWER]: Refusing update from remote peer f1e583d03c7442b488f94d519db8990a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:01.114089 25448 consensus_queue.cc:1035] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ebb8cb294b4b0bbe1ca847373af8ac" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 35515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:01.114514 25445 consensus_queue.cc:1035] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56de784dae9747df9c0f905debfee605" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:01.124188 25451 mvcc.cc:205] Tried to move back new op lower bound from 7334698111426949120 to 7334698111091576832. Current Snapshot: MvccSnapshot[applied={T|T < 7334698111426949120}]
I20260929 16:03:01.125973 25452 mvcc.cc:205] Tried to move back new op lower bound from 7334698111426949120 to 7334698111091576832. Current Snapshot: MvccSnapshot[applied={T|T < 7334698111426949120}]
I20260929 16:03:01.142277 24960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:03:01.142619 24960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:03:01.146711 24960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2b644931304411587d860ad2bef1142: Sending DeleteTablet for 3 replicas of tablet 25f76502de1049249228677086fb9497
I20260929 16:03:01.147995 25219 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f76502de1049249228677086fb9497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:01 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:52882
I20260929 16:03:01.148267 25085 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f76502de1049249228677086fb9497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:01 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:40184
I20260929 16:03:01.149272 25353 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f76502de1049249228677086fb9497 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:01 UTC) from {username='kudu', principal='kudu/127.5.117.62@KRBTEST.COM'} at 127.0.0.1:50388
I20260929 16:03:01.150048 25461 tablet_replica.cc:330] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: stopping tablet replica
I20260929 16:03:01.150697 25461 raft_consensus.cc:2229] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:03:01.151482 25461 raft_consensus.cc:2258] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:01.151686 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25022
I20260929 16:03:01.154681 25461 ts_tablet_manager.cc:1905] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:01.156111 25460 tablet_replica.cc:330] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: stopping tablet replica
I20260929 16:03:01.156996 25460 raft_consensus.cc:2229] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:03:01.158046 25460 raft_consensus.cc:2258] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:01.160712 25460 ts_tablet_manager.cc:1905] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:01.165923 25461 ts_tablet_manager.cc:1918] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:01.166131 25461 log.cc:1196] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/25f76502de1049249228677086fb9497
I20260929 16:03:01.166781 25461 ts_tablet_manager.cc:1939] T 25f76502de1049249228677086fb9497 P 61ebb8cb294b4b0bbe1ca847373af8ac: Deleting consensus metadata
Sep 29 16:03:01 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.1@KRBTEST.COM
I20260929 16:03:01.169149 24945 catalog_manager.cc:4841] TS 61ebb8cb294b4b0bbe1ca847373af8ac (127.5.117.1:35515): tablet 25f76502de1049249228677086fb9497 (table test-table [id=2ab2cc6469514067a10ab46410a62b3b]) successfully deleted
I20260929 16:03:01.170137 25462 tablet_replica.cc:330] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: stopping tablet replica
I20260929 16:03:01.170706 25462 raft_consensus.cc:2229] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:03:01.171296 25462 raft_consensus.cc:2258] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:01.173537 25460 ts_tablet_manager.cc:1918] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:01.173946 25460 log.cc:1196] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/25f76502de1049249228677086fb9497
I20260929 16:03:01.173975 25462 ts_tablet_manager.cc:1905] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:01.174857 25460 ts_tablet_manager.cc:1939] T 25f76502de1049249228677086fb9497 P f1e583d03c7442b488f94d519db8990a: Deleting consensus metadata
I20260929 16:03:01.177636 24948 catalog_manager.cc:4841] TS f1e583d03c7442b488f94d519db8990a (127.5.117.2:44973): tablet 25f76502de1049249228677086fb9497 (table test-table [id=2ab2cc6469514067a10ab46410a62b3b]) successfully deleted
I20260929 16:03:01.180863 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.153667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.5.117.1:35515)
0929 16:03:01.153928 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.153940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.156349 (+ 2409us) server_negotiation.cc:407] Connection header received
0929 16:03:01.156658 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.156665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.156766 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.156856 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:01.159188 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.160458 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.162669 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.163526 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.163641 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.170989 (+ 7348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:01.171016 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:01.171038 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:01.171113 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:01.176590 (+ 5477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.177843 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.177855 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.177858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.178084 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.178677 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.178682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.178685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.179293 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:01.179838 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.180138 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.180398 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:01.181542 25145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:01.182086 25462 ts_tablet_manager.cc:1918] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:01.182251 25462 log.cc:1196] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/25f76502de1049249228677086fb9497
I20260929 16:03:01.182899 25462 ts_tablet_manager.cc:1939] T 25f76502de1049249228677086fb9497 P 56de784dae9747df9c0f905debfee605: Deleting consensus metadata
I20260929 16:03:01.184953 24945 catalog_manager.cc:4841] TS 56de784dae9747df9c0f905debfee605 (127.5.117.3:43399): tablet 25f76502de1049249228677086fb9497 (table test-table [id=2ab2cc6469514067a10ab46410a62b3b]) successfully deleted
I20260929 16:03:01.301615 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25022
I20260929 16:03:01.323395 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25156
Sep 29 16:03:01 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.2@KRBTEST.COM
I20260929 16:03:01.344723 25433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.324461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.5.117.2:44973)
0929 16:03:01.324688 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.324695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.324720 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:03:01.324902 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.324909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.325024 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.325147 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:01.327044 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.328408 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.330002 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.330795 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.330952 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.336044 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:01.336095 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:01.336126 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:01.336200 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:01.341116 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.342072 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.342086 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.342090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.342268 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.342730 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.342735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.342738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.343298 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:01.343672 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.343940 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.344258 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:03:01.345750 25279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:01.472923 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25156
I20260929 16:03:01.498045 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25290
Sep 29 16:03:01 dist-test-slave-b69v krb5kdc[24916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.3@KRBTEST.COM
I20260929 16:03:01.515622 25434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:01.499076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.5.117.3:43399)
0929 16:03:01.499295 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:01.499302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:01.499323 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:03:01.499446 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:01.499452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:01.499552 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:01.499678 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:01.501221 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.502038 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.503441 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:01.503857 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:01.503989 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:01.509404 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:01.509423 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:01.509444 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:01.509483 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:01.512582 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.513316 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.513324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.513326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.513474 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:01.513850 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:01.513857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:01.513861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:01.514284 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:01.514615 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:01.514981 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 16:03:01.515253 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:01.516490 25413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:01.614295 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25290
I20260929 16:03:01.633397 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 24929
I20260929 16:03:01.634702 24991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:01.755427 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 24929
2026-09-29T16:03:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:01Z Disabled control of system clock
I20260929 16:03:01.852247 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46471
--webserver_interface=127.5.117.62
--webserver_port=0
--builtin_ntp_servers=127.5.117.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:03:02.236716 25471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:02.237071 25471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:02.237243 25471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:02.244823 25471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:02.244943 25471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:02.245052 25471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:02.245162 25471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:02.245240 25471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:02.254980 25471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:02.256670 25471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:02.258895 25471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:02.272794 25476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.272701 25479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.272732 25477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:02.274048 25471 server_base.cc:1034] running on GCE node
I20260929 16:03:02.275096 25471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:02.277037 25471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:02.278342 25471 hybrid_clock.cc:648] HybridClock initialized: now 1790697782278283 us; error 68 us; skew 500 ppm
I20260929 16:03:02.278843 25471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:03:02.282409 25471 webserver.cc:460] Webserver started at http://127.5.117.62:34057/ using document root <none> and password file <none>
I20260929 16:03:02.283319 25471 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:02.283471 25471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:02.283875 25471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:02.288489 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "5c3ebb8628344abe9ca47577de0bd69a"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "3dc4cf9cc2d71c4bcafed08c2ff322bb"
server_key_iv: "c4c25c0c9a0e81a3cbcebf0e549bde66"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.289475 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "5c3ebb8628344abe9ca47577de0bd69a"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "3dc4cf9cc2d71c4bcafed08c2ff322bb"
server_key_iv: "c4c25c0c9a0e81a3cbcebf0e549bde66"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.296738 25471 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 16:03:02.301199 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:02.302991 25471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260929 16:03:02.303201 25471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "5c3ebb8628344abe9ca47577de0bd69a"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "3dc4cf9cc2d71c4bcafed08c2ff322bb"
server_key_iv: "c4c25c0c9a0e81a3cbcebf0e549bde66"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.303483 25471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:02.330597 25471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:02.331673 25471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:02.332026 25471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:02.355087 25471 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46471
I20260929 16:03:02.355115 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46471 every 8 connection(s)
I20260929 16:03:02.356743 25471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:02.361253 25537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:02.362143 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25471
I20260929 16:03:02.362349 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:02.362881 5588 external_mini_cluster.cc:1428] Setting key 17eee5b6e8fd3661e0d4faa605d90891
I20260929 16:03:02.377856 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: Bootstrap starting.
I20260929 16:03:02.379761 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:02.365157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.5.117.62:46471)
0929 16:03:02.366363 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:02.366378 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:02.366437 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:03:02.368784 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:02.368817 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:02.369519 (+ 702us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:02.370022 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:02.370644 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.372736 (+ 2092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.374164 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.375289 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.375471 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:02.376743 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:02.376758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:02.376777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:02.377559 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:02.377696 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:02.377725 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:02.377864 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:02.378013 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:03:02.378362 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":382,"threads_started":1}
I20260929 16:03:02.382088 25537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:02.383332 25537 log.cc:824] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:02.386471 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: No bootstrap required, opened a new log
I20260929 16:03:02.395870 25537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } }
I20260929 16:03:02.396422 25537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:02.396525 25537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3ebb8628344abe9ca47577de0bd69a, State: Initialized, Role: FOLLOWER
I20260929 16:03:02.397075 25537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } }
I20260929 16:03:02.397297 25537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:02.397403 25537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:02.397542 25537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:02.400368 25537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } }
I20260929 16:03:02.400956 25537 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3ebb8628344abe9ca47577de0bd69a; no voters:
I20260929 16:03:02.401559 25537 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:02.401751 25542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:02.402515 25542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [term 1 LEADER]: Becoming Leader. State: Replica: 5c3ebb8628344abe9ca47577de0bd69a, State: Running, Role: LEADER
I20260929 16:03:02.403164 25542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } }
I20260929 16:03:02.403988 25537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:02.406015 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3ebb8628344abe9ca47577de0bd69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } } }
I20260929 16:03:02.406450 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:02.408205 25547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:02.410084 25544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3ebb8628344abe9ca47577de0bd69a. Latest consensus state: current_term: 1 leader_uuid: "5c3ebb8628344abe9ca47577de0bd69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3ebb8628344abe9ca47577de0bd69a" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46471 } } }
I20260929 16:03:02.410410 25544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:02.414901 25547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:02.424489 25547 catalog_manager.cc:1347] Generated new cluster ID: 29063430b74b423e867585e9afb46609
I20260929 16:03:02.424616 25547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:02.434580 25547 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:02.436316 25547 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:02.448660 25547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: Generated new TSK 0
I20260929 16:03:02.449913 25547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:02.466399 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.1:0
--local_ip_for_outbound_sockets=127.5.117.1
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--builtin_ntp_servers=127.5.117.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:03:02.885546 25561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:02.885826 25561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:02.885927 25561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:02.893105 25561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:02.893199 25561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:02.893323 25561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.1
I20260929 16:03:02.901883 25561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.117.1
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.1
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:02.903163 25561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:02.905287 25561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:02.919219 25569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.919329 25566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.919304 25567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:02.920329 25561 server_base.cc:1034] running on GCE node
I20260929 16:03:02.920869 25561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:02.921721 25561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:02.922960 25561 hybrid_clock.cc:648] HybridClock initialized: now 1790697782922937 us; error 43 us; skew 500 ppm
I20260929 16:03:02.923281 25561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:03:02.925616 25561 webserver.cc:460] Webserver started at http://127.5.117.1:44931/ using document root <none> and password file <none>
I20260929 16:03:02.926326 25561 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:02.926456 25561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:02.926797 25561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:02.929893 25561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data/instance:
uuid: "bdb70cc3f9c646d89691f67bc23eb527"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "c0423d0076a677b3409e8cd17a55484a"
server_key_iv: "4ecea5b7906a8abee7041e02fde574dc"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.930703 25561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance:
uuid: "bdb70cc3f9c646d89691f67bc23eb527"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "c0423d0076a677b3409e8cd17a55484a"
server_key_iv: "4ecea5b7906a8abee7041e02fde574dc"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.935987 25561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:03:02.939512 25575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:02.940953 25561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:02.941146 25561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal
uuid: "bdb70cc3f9c646d89691f67bc23eb527"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-b69v"
server_key: "c0423d0076a677b3409e8cd17a55484a"
server_key_iv: "4ecea5b7906a8abee7041e02fde574dc"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.941366 25561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:02.971163 25561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:02.972038 25561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:02.972373 25561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:02.973405 25561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:03:02.975006 25561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:03:02.975136 25561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:02.975261 25561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:03:02.975358 25561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:03.011190 25561 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.1:36073
I20260929 16:03:03.011276 25687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.1:36073 every 8 connection(s)
I20260929 16:03:03.012632 25561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/data/info.pb
I20260929 16:03:03.021339 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25561
I20260929 16:03:03.021567 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal/instance
I20260929 16:03:03.022090 5588 external_mini_cluster.cc:1428] Setting key ea68172a5c8c5d996ab4a6fb507f6260
I20260929 16:03:03.025149 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.2:0
--local_ip_for_outbound_sockets=127.5.117.2
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--builtin_ntp_servers=127.5.117.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:03:03.026476 25690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:03.015477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.1:50181 (local address 127.5.117.62:46471)
0929 16:03:03.016017 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:03.016022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:03.017129 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 16:03:03.019082 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:03.019089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:03.019188 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:03.019320 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:03.020537 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.021666 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.023403 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.023951 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.024113 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:03.025597 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:03.025604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:03.025615 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:03.025633 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:03.025675 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:03.025687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:03.025785 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:03.026016 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:03:03.026115 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":289,"threads_started":1}
I20260929 16:03:03.027446 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:03.015995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46471 (local address 127.5.117.1:50181)
0929 16:03:03.016975 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:03.017011 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:03.018788 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:03.019591 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:03.019599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:03.019683 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:03.020355 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:03.020377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.021848 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.021852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:03.023269 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:03.023282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.024226 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.024233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:03.024465 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:03.025237 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:03:03.025240 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:03.025302 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:03:03.025408 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:03.025784 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:03.025797 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:03.025807 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:03:03.025986 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:03:03.026354 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":308,"threads_started":1}
I20260929 16:03:03.029048 25688 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46471
I20260929 16:03:03.029353 25688 heartbeater.cc:464] Registering TS with master...
I20260929 16:03:03.030130 25688 heartbeater.cc:511] Master 127.5.117.62:46471 requested a full tablet report, sending...
I20260929 16:03:03.032274 25502 ts_manager.cc:194] Registered new tserver with Master: bdb70cc3f9c646d89691f67bc23eb527 (127.5.117.1:36073)
I20260929 16:03:03.033844 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.1:50181
W20260929 16:03:03.405306 25692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:03.405685 25692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:03.405786 25692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:03.414543 25692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:03.414701 25692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:03.414834 25692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.2
I20260929 16:03:03.425171 25692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.117.2
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.2
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:03.426841 25692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:03.428980 25692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:03.444764 25697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:03.444721 25698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:03.445562 25692 server_base.cc:1034] running on GCE node
W20260929 16:03:03.445650 25700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:03.446434 25692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:03.447613 25692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:03.448858 25692 hybrid_clock.cc:648] HybridClock initialized: now 1790697783448811 us; error 86 us; skew 500 ppm
I20260929 16:03:03.449235 25692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:03:03.452188 25692 webserver.cc:460] Webserver started at http://127.5.117.2:44513/ using document root <none> and password file <none>
I20260929 16:03:03.453074 25692 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:03.453219 25692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:03.453593 25692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:03.457106 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data/instance:
uuid: "beda5726f02a4fa9a31de782b0b64a87"
format_stamp: "Formatted at 2026-09-29 16:03:03 on dist-test-slave-b69v"
server_key: "115eb033519ce6cec3108e651996db44"
server_key_iv: "cd0701102bbcef52b864aa69b5151014"
server_key_version: "encryptionkey@0"
I20260929 16:03:03.457988 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance:
uuid: "beda5726f02a4fa9a31de782b0b64a87"
format_stamp: "Formatted at 2026-09-29 16:03:03 on dist-test-slave-b69v"
server_key: "115eb033519ce6cec3108e651996db44"
server_key_iv: "cd0701102bbcef52b864aa69b5151014"
server_key_version: "encryptionkey@0"
I20260929 16:03:03.463766 25692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:03:03.467645 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:03.469354 25692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:03.469578 25692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal
uuid: "beda5726f02a4fa9a31de782b0b64a87"
format_stamp: "Formatted at 2026-09-29 16:03:03 on dist-test-slave-b69v"
server_key: "115eb033519ce6cec3108e651996db44"
server_key_iv: "cd0701102bbcef52b864aa69b5151014"
server_key_version: "encryptionkey@0"
I20260929 16:03:03.469844 25692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:03.505565 25692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:03.506644 25692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:03.506979 25692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:03.508368 25692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:03:03.510233 25692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:03:03.510385 25692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:03.510524 25692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:03:03.510617 25692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:03.551101 25692 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.2:34827
I20260929 16:03:03.551236 25818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.2:34827 every 8 connection(s)
I20260929 16:03:03.552826 25692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/data/info.pb
I20260929 16:03:03.560926 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25692
I20260929 16:03:03.561142 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal/instance
I20260929 16:03:03.561656 5588 external_mini_cluster.cc:1428] Setting key 3b749a197bb6cce4e93aa44f33bcf16e
I20260929 16:03:03.564774 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.117.3:0
--local_ip_for_outbound_sockets=127.5.117.3
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--builtin_ntp_servers=127.5.117.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:03:03.567220 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:03.555919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:53659 (local address 127.5.117.62:46471)
0929 16:03:03.556486 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:03.556502 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:03.557798 (+ 1296us) server_negotiation.cc:407] Connection header received
0929 16:03:03.559804 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:03.559812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:03.559914 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:03.560084 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:03.561396 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.562233 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.564093 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.564502 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.564641 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:03.566270 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:03.566275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:03.566279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:03.566298 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:03.566343 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:03.566356 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:03.566461 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:03.566698 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:03:03.566810 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260929 16:03:03.568471 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:03.556633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46471 (local address 127.5.117.2:53659)
0929 16:03:03.557623 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:03.557667 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:03.559516 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:03.560369 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:03.560378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:03.560486 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:03.561206 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:03.561233 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.562380 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.562385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:03.563919 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:03.563937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:03.564732 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:03.564738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:03.565100 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:03.565912 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:03:03.565914 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:03.565961 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:03:03.566053 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:03.566457 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:03.566470 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:03.566481 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:03:03.566688 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 16:03:03.567147 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":321,"threads_started":1}
I20260929 16:03:03.570075 25819 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46471
I20260929 16:03:03.570459 25819 heartbeater.cc:464] Registering TS with master...
I20260929 16:03:03.571349 25819 heartbeater.cc:511] Master 127.5.117.62:46471 requested a full tablet report, sending...
I20260929 16:03:03.573112 25502 ts_manager.cc:194] Registered new tserver with Master: beda5726f02a4fa9a31de782b0b64a87 (127.5.117.2:34827)
I20260929 16:03:03.574117 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.2:53659
W20260929 16:03:03.957093 25823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:03.957440 25823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:03.957537 25823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:03.965241 25823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:03.965371 25823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:03.965503 25823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.117.3
I20260929 16:03:03.976410 25823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.117.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.117.3
--webserver_port=0
--tserver_master_addrs=127.5.117.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.117.3
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:03.978194 25823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:03.980414 25823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:03.996759 25829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:03.996804 25828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:03.996706 25831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:03.998708 25823 server_base.cc:1034] running on GCE node
I20260929 16:03:03.999464 25823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:04.000625 25823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:04.001890 25823 hybrid_clock.cc:648] HybridClock initialized: now 1790697784001848 us; error 54 us; skew 500 ppm
I20260929 16:03:04.002281 25823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:03:04.005148 25823 webserver.cc:460] Webserver started at http://127.5.117.3:40575/ using document root <none> and password file <none>
I20260929 16:03:04.005975 25823 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:04.006070 25823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:04.006395 25823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:04.009910 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data/instance:
uuid: "37442d9e25104f6d81715dace31978b6"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-b69v"
server_key: "d71411ba396458ed45b6b3126562b24c"
server_key_iv: "cb20883ef4a5544c6e4bc44a40f0ef8f"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.010974 25823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance:
uuid: "37442d9e25104f6d81715dace31978b6"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-b69v"
server_key: "d71411ba396458ed45b6b3126562b24c"
server_key_iv: "cb20883ef4a5544c6e4bc44a40f0ef8f"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.018872 25823 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 16:03:04.023401 25837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:04.025152 25823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:03:04.025341 25823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal
uuid: "37442d9e25104f6d81715dace31978b6"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-b69v"
server_key: "d71411ba396458ed45b6b3126562b24c"
server_key_iv: "cb20883ef4a5544c6e4bc44a40f0ef8f"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.025604 25823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:04.036926 25688 heartbeater.cc:503] Master 127.5.117.62:46471 was elected leader, sending a full tablet report...
I20260929 16:03:04.040735 25823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:04.041703 25823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:04.042024 25823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:04.043190 25823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:03:04.044965 25823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:03:04.045099 25823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:04.045234 25823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:03:04.045323 25823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:04.089936 25823 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.3:43897
I20260929 16:03:04.089979 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.3:43897 every 8 connection(s)
I20260929 16:03:04.091621 25823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/data/info.pb
I20260929 16:03:04.094116 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 25823
I20260929 16:03:04.094332 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal/instance
I20260929 16:03:04.094897 5588 external_mini_cluster.cc:1428] Setting key fd3e3b90134e72c76f9c99384f489866
I20260929 16:03:04.107080 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.096303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.3:52039 (local address 127.5.117.62:46471)
0929 16:03:04.100453 (+ 4150us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.100459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.100475 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:03:04.100527 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.100532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.100599 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.100695 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.101840 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.102664 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.104378 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.104809 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.104959 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.106311 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.106317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.106320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.106339 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.106382 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.106394 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.106464 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.106665 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.106755 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4022,"thread_start_us":3914,"threads_started":1}
I20260929 16:03:04.108254 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.095705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46471 (local address 127.5.117.3:52039)
0929 16:03:04.097101 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.097161 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.099290 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.100945 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.100953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.101047 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:04.101692 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.101709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.102796 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.102800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.104226 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.104237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.105011 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.105016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.105220 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.106000 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:03:04.106003 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:04.106044 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:03:04.106144 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:04.106478 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:04.106485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:04.106491 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.106639 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.106960 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":486,"threads_started":1}
I20260929 16:03:04.109870 25950 heartbeater.cc:347] Connected to a master server at 127.5.117.62:46471
I20260929 16:03:04.110255 25950 heartbeater.cc:464] Registering TS with master...
I20260929 16:03:04.111301 25950 heartbeater.cc:511] Master 127.5.117.62:46471 requested a full tablet report, sending...
I20260929 16:03:04.112814 25502 ts_manager.cc:194] Registered new tserver with Master: 37442d9e25104f6d81715dace31978b6 (127.5.117.3:43897)
I20260929 16:03:04.113802 25502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.117.3:52039
I20260929 16:03:04.123430 5588 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:03:04.134137 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.127992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.5.117.62:46471)
0929 16:03:04.128223 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.128229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.128679 (+ 450us) server_negotiation.cc:407] Connection header received
0929 16:03:04.128833 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.128838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.128908 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.129012 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.129529 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.130330 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.131818 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.132335 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.132435 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.133511 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.133516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.133519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.133539 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.133574 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.133599 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.133669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.133792 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.133873 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 16:03:04.138415 25502 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:03:04.146379 25502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:03:04.175669 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.161415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43897 (local address 127.0.0.1:58482)
0929 16:03:04.163201 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.163240 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.163380 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.164622 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.164633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.164676 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:04.165229 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.165246 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.167852 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.167861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.171081 (+ 3220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.171103 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.174266 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.174276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.174573 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.174649 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.174810 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.175120 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1680,"thread_start_us":1527,"threads_started":1}
I20260929 16:03:04.177112 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.162152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.5.117.3:43897)
0929 16:03:04.162950 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.162963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.163520 (+ 557us) server_negotiation.cc:407] Connection header received
0929 16:03:04.163654 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.163666 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.163976 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.164214 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:04.165517 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.167622 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.171380 (+ 3758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.173519 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.173675 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.174094 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.176292 (+ 2198us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.176562 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":450,"threads_started":1}
I20260929 16:03:04.182286 25885 tablet_service.cc:1467] Processing CreateTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (DEFAULT_TABLE table=test-table [id=39762072ac804669b24d07af139590f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:04.184027 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.160569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.2:34827 (local address 127.0.0.1:46236)
0929 16:03:04.161449 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.161504 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.161708 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.177223 (+ 15515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.177228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.177253 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:04.177652 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.177664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.179752 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.179759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.181778 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.181790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.183131 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.183137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.183332 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.183373 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.183508 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.183650 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":6,"thread_start_us":329,"threads_started":1}
I20260929 16:03:04.184131 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.160847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.5.117.2:34827)
0929 16:03:04.176179 (+ 15332us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.176202 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.176241 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:03:04.176366 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.176374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.176704 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.177890 (+ 1186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:04.178151 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.179588 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.181981 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.182925 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.183098 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.183370 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.183552 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.183724 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15086,"thread_start_us":16009,"threads_started":1}
I20260929 16:03:04.184217 25885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a1ce6253b14a3a8ff8171db2d4ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:04.187079 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.163777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36073 (local address 127.0.0.1:38216)
0929 16:03:04.175481 (+ 11704us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.175517 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.175686 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.178694 (+ 3008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.178699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.178730 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:04.179143 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.179155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.181180 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.181185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.183401 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.183416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.186274 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.186283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.186472 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.186505 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.186595 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.186716 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11512,"mutex_wait_us":25,"thread_start_us":12009,"threads_started":1}
I20260929 16:03:04.187289 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.163931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.5.117.1:36073)
0929 16:03:04.177993 (+ 14062us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.178002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.178035 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:03:04.178149 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.178157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.178434 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.178660 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:04.179345 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.181029 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.183641 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.186059 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.186221 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.186471 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.186719 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.186887 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13851,"thread_start_us":13670,"threads_started":1}
I20260929 16:03:04.188015 25754 tablet_service.cc:1467] Processing CreateTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (DEFAULT_TABLE table=test-table [id=39762072ac804669b24d07af139590f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:04.189779 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a1ce6253b14a3a8ff8171db2d4ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:04.192119 25623 tablet_service.cc:1467] Processing CreateTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (DEFAULT_TABLE table=test-table [id=39762072ac804669b24d07af139590f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:03:04.193707 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a1ce6253b14a3a8ff8171db2d4ed01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:04.198344 25969 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Bootstrap starting.
I20260929 16:03:04.202967 25969 tablet_bootstrap.cc:654] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:04.204519 25969 log.cc:824] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:04.206003 25970 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Bootstrap starting.
I20260929 16:03:04.209542 25972 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Bootstrap starting.
I20260929 16:03:04.210472 25970 tablet_bootstrap.cc:654] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:04.210542 25969 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: No bootstrap required, opened a new log
I20260929 16:03:04.210860 25969 ts_tablet_manager.cc:1397] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260929 16:03:04.211970 25970 log.cc:824] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:04.213263 25972 tablet_bootstrap.cc:654] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:04.214819 25972 log.cc:824] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:04.216890 25970 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: No bootstrap required, opened a new log
I20260929 16:03:04.217211 25970 ts_tablet_manager.cc:1397] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 16:03:04.217604 25972 tablet_bootstrap.cc:492] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: No bootstrap required, opened a new log
I20260929 16:03:04.217871 25972 ts_tablet_manager.cc:1397] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:03:04.222429 25969 raft_consensus.cc:348] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.223002 25969 raft_consensus.cc:374] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:04.223114 25969 raft_consensus.cc:729] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37442d9e25104f6d81715dace31978b6, State: Initialized, Role: FOLLOWER
I20260929 16:03:04.224011 25969 consensus_queue.cc:260] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.225764 25950 heartbeater.cc:503] Master 127.5.117.62:46471 was elected leader, sending a full tablet report...
I20260929 16:03:04.226403 25972 raft_consensus.cc:348] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.226922 25972 raft_consensus.cc:374] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:04.227041 25972 raft_consensus.cc:729] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdb70cc3f9c646d89691f67bc23eb527, State: Initialized, Role: FOLLOWER
I20260929 16:03:04.227471 25969 ts_tablet_manager.cc:1428] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260929 16:03:04.227873 25972 consensus_queue.cc:260] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.229077 25970 raft_consensus.cc:348] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.229547 25970 raft_consensus.cc:374] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:04.229645 25970 raft_consensus.cc:729] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beda5726f02a4fa9a31de782b0b64a87, State: Initialized, Role: FOLLOWER
I20260929 16:03:04.229820 25972 ts_tablet_manager.cc:1428] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260929 16:03:04.230357 25970 consensus_queue.cc:260] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.233870 25819 heartbeater.cc:503] Master 127.5.117.62:46471 was elected leader, sending a full tablet report...
I20260929 16:03:04.234517 25970 ts_tablet_manager.cc:1428] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260929 16:03:04.265584 25689 tablet.cc:2369] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:03:04.305730 25820 tablet.cc:2369] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:03:04.343899 25951 tablet.cc:2369] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:04.346092 25977 raft_consensus.cc:482] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:03:04.346444 25977 raft_consensus.cc:504] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.348528 25977 leader_election.cc:288] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37442d9e25104f6d81715dace31978b6 (127.5.117.3:43897), bdb70cc3f9c646d89691f67bc23eb527 (127.5.117.1:36073)
I20260929 16:03:04.356314 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.350056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:50043 (local address 127.5.117.1:36073)
0929 16:03:04.350284 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.350291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.350415 (+ 124us) server_negotiation.cc:407] Connection header received
0929 16:03:04.350592 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.350600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.350710 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.350821 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:04.351455 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.352302 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.354256 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.355231 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.355402 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.355521 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.355864 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.356023 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:03:04.356374 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.349676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.1:36073 (local address 127.5.117.2:50043)
0929 16:03:04.350303 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.350323 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.350458 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.350840 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.350843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.350861 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:04.351287 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.351300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.352493 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.352497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.354023 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.354036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.355431 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.355437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.355637 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.355701 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.355816 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.355976 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":4,"thread_start_us":346,"threads_started":1}
I20260929 16:03:04.356556 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.349012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.117.2:49855 (local address 127.5.117.3:43897)
0929 16:03:04.349293 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.349305 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.349783 (+ 478us) server_negotiation.cc:407] Connection header received
0929 16:03:04.350080 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.350087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.350183 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.350256 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:03:04.351242 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.352154 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.354204 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.355458 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.355579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.355696 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.356130 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.356268 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:04.357350 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.348960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.3:43897 (local address 127.5.117.2:49855)
0929 16:03:04.349599 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:04.349646 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:04.349849 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:04.350541 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:04.350545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:04.350574 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:03:04.351072 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.351086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.352289 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.352294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.353995 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:04.354007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.355652 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.355658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:04.355913 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.355945 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:03:04.356854 (+ 909us) client_negotiation.cc:240] Negotiation successful
0929 16:03:04.356993 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":74,"thread_start_us":326,"threads_started":1}
I20260929 16:03:04.357724 25643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a1ce6253b14a3a8ff8171db2d4ed01" candidate_uuid: "beda5726f02a4fa9a31de782b0b64a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb70cc3f9c646d89691f67bc23eb527" is_pre_election: true
I20260929 16:03:04.358435 25643 raft_consensus.cc:2454] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beda5726f02a4fa9a31de782b0b64a87 in term 0.
I20260929 16:03:04.358547 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a1ce6253b14a3a8ff8171db2d4ed01" candidate_uuid: "beda5726f02a4fa9a31de782b0b64a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37442d9e25104f6d81715dace31978b6" is_pre_election: true
I20260929 16:03:04.359108 25905 raft_consensus.cc:2454] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate beda5726f02a4fa9a31de782b0b64a87 in term 0.
I20260929 16:03:04.359275 25708 leader_election.cc:302] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb70cc3f9c646d89691f67bc23eb527, beda5726f02a4fa9a31de782b0b64a87; no voters:
I20260929 16:03:04.359882 25977 raft_consensus.cc:2781] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:03:04.360097 25977 raft_consensus.cc:482] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:03:04.360255 25977 raft_consensus.cc:3037] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:04.362771 25977 raft_consensus.cc:504] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.363466 25977 leader_election.cc:288] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [CANDIDATE]: Term 1 election: Requested vote from peers 37442d9e25104f6d81715dace31978b6 (127.5.117.3:43897), bdb70cc3f9c646d89691f67bc23eb527 (127.5.117.1:36073)
I20260929 16:03:04.363950 25905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a1ce6253b14a3a8ff8171db2d4ed01" candidate_uuid: "beda5726f02a4fa9a31de782b0b64a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37442d9e25104f6d81715dace31978b6"
I20260929 16:03:04.364029 25643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a1ce6253b14a3a8ff8171db2d4ed01" candidate_uuid: "beda5726f02a4fa9a31de782b0b64a87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdb70cc3f9c646d89691f67bc23eb527"
I20260929 16:03:04.364214 25905 raft_consensus.cc:3037] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:04.364291 25643 raft_consensus.cc:3037] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:04.367838 25643 raft_consensus.cc:2454] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beda5726f02a4fa9a31de782b0b64a87 in term 1.
I20260929 16:03:04.367902 25905 raft_consensus.cc:2454] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate beda5726f02a4fa9a31de782b0b64a87 in term 1.
I20260929 16:03:04.368314 25708 leader_election.cc:302] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdb70cc3f9c646d89691f67bc23eb527, beda5726f02a4fa9a31de782b0b64a87; no voters:
I20260929 16:03:04.368639 25977 raft_consensus.cc:2781] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:04.369179 25977 raft_consensus.cc:686] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 1 LEADER]: Becoming Leader. State: Replica: beda5726f02a4fa9a31de782b0b64a87, State: Running, Role: LEADER
I20260929 16:03:04.369841 25977 consensus_queue.cc:237] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } }
I20260929 16:03:04.375033 25502 catalog_manager.cc:5495] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 reported cstate change: term changed from 0 to 1, leader changed from <none> to beda5726f02a4fa9a31de782b0b64a87 (127.5.117.2). New cstate: current_term: 1 leader_uuid: "beda5726f02a4fa9a31de782b0b64a87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beda5726f02a4fa9a31de782b0b64a87" member_type: VOTER last_known_addr { host: "127.5.117.2" port: 34827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260929 16:03:04.427129 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.421358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.5.117.2:34827)
0929 16:03:04.421573 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.421577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.421595 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:03:04.421706 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.421710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.421818 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.422022 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:04.422352 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.423119 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.424631 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.425124 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.425227 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.425828 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:04.426579 (+ 751us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:04.426698 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.426796 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.426882 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:03:04.437198 25643 raft_consensus.cc:1261] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 1 FOLLOWER]: Refusing update from remote peer beda5726f02a4fa9a31de782b0b64a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:04.437198 25905 raft_consensus.cc:1261] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 1 FOLLOWER]: Refusing update from remote peer beda5726f02a4fa9a31de782b0b64a87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:04.438306 25980 consensus_queue.cc:1035] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37442d9e25104f6d81715dace31978b6" member_type: VOTER last_known_addr { host: "127.5.117.3" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:04.438710 25977 consensus_queue.cc:1035] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdb70cc3f9c646d89691f67bc23eb527" member_type: VOTER last_known_addr { host: "127.5.117.1" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:04.452302 25983 mvcc.cc:205] Tried to move back new op lower bound from 7334698125042081792 to 7334698124783423488. Current Snapshot: MvccSnapshot[applied={T|T < 7334698125042081792}]
I20260929 16:03:04.465586 25502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:03:04.465912 25502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:03:04.469903 25502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3ebb8628344abe9ca47577de0bd69a: Sending DeleteTablet for 3 replicas of tablet 47a1ce6253b14a3a8ff8171db2d4ed01
I20260929 16:03:04.471186 25885 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:04 UTC) from {username='slave'} at 127.0.0.1:58482
I20260929 16:03:04.471575 25623 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:04 UTC) from {username='slave'} at 127.0.0.1:38216
I20260929 16:03:04.472333 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a1ce6253b14a3a8ff8171db2d4ed01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:03:04 UTC) from {username='slave'} at 127.0.0.1:46236
I20260929 16:03:04.473415 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25561
I20260929 16:03:04.487355 25994 tablet_replica.cc:330] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: stopping tablet replica
I20260929 16:03:04.489261 25993 tablet_replica.cc:330] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: stopping tablet replica
I20260929 16:03:04.489282 25995 tablet_replica.cc:330] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: stopping tablet replica
I20260929 16:03:04.489677 25993 raft_consensus.cc:2229] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:03:04.488000 25994 raft_consensus.cc:2229] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:03:04.490286 25993 raft_consensus.cc:2258] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:04.490612 25994 raft_consensus.cc:2258] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:04.490832 25995 raft_consensus.cc:2229] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:03:04.493208 25993 ts_tablet_manager.cc:1905] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:04.493223 25994 ts_tablet_manager.cc:1905] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:04.495461 25995 raft_consensus.cc:2258] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:03:04.497581 25995 ts_tablet_manager.cc:1905] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:03:04.501083 25993 ts_tablet_manager.cc:1918] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:04.501113 25994 ts_tablet_manager.cc:1918] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:04.501288 25993 log.cc:1196] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-2/wal/wals/47a1ce6253b14a3a8ff8171db2d4ed01
I20260929 16:03:04.501447 25994 log.cc:1196] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-0/wal/wals/47a1ce6253b14a3a8ff8171db2d4ed01
I20260929 16:03:04.501910 25993 ts_tablet_manager.cc:1939] T 47a1ce6253b14a3a8ff8171db2d4ed01 P 37442d9e25104f6d81715dace31978b6: Deleting consensus metadata
I20260929 16:03:04.502249 25994 ts_tablet_manager.cc:1939] T 47a1ce6253b14a3a8ff8171db2d4ed01 P bdb70cc3f9c646d89691f67bc23eb527: Deleting consensus metadata
I20260929 16:03:04.504041 25487 catalog_manager.cc:4841] TS 37442d9e25104f6d81715dace31978b6 (127.5.117.3:43897): tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (table test-table [id=39762072ac804669b24d07af139590f4]) successfully deleted
I20260929 16:03:04.504281 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.475830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.5.117.1:36073)
0929 16:03:04.476108 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.476118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.488117 (+ 11999us) server_negotiation.cc:407] Connection header received
0929 16:03:04.488746 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.488755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.488907 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.489030 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.489739 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.491004 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.492919 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.493546 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.493671 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.501627 (+ 7956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.501641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.501667 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.502583 (+ 916us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.502651 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.502678 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.502790 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.503466 (+ 676us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.503599 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:03:04.504469 25487 catalog_manager.cc:4841] TS bdb70cc3f9c646d89691f67bc23eb527 (127.5.117.1:36073): tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (table test-table [id=39762072ac804669b24d07af139590f4]) successfully deleted
I20260929 16:03:04.505061 25683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:04.508880 25995 ts_tablet_manager.cc:1918] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:03:04.509109 25995 log.cc:1196] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Deleting WAL directory at /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697441705944-5588-0/minicluster-data/ts-1/wal/wals/47a1ce6253b14a3a8ff8171db2d4ed01
I20260929 16:03:04.509654 25995 ts_tablet_manager.cc:1939] T 47a1ce6253b14a3a8ff8171db2d4ed01 P beda5726f02a4fa9a31de782b0b64a87: Deleting consensus metadata
I20260929 16:03:04.511097 25489 catalog_manager.cc:4841] TS beda5726f02a4fa9a31de782b0b64a87 (127.5.117.2:34827): tablet 47a1ce6253b14a3a8ff8171db2d4ed01 (table test-table [id=39762072ac804669b24d07af139590f4]) successfully deleted
I20260929 16:03:04.624815 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25561
I20260929 16:03:04.648204 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25692
I20260929 16:03:04.657917 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.649230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.5.117.2:34827)
0929 16:03:04.649455 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.649461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.649486 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:03:04.649695 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.649701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.649829 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.649973 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.650363 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.651660 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.653710 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.654461 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.654587 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.655753 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.655764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.655787 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.656999 (+ 1212us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.657080 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.657116 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.657236 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.657327 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.657452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:03:04.658735 25814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:04.789568 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25692
I20260929 16:03:04.813271 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25823
I20260929 16:03:04.820514 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.814215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.5.117.3:43897)
0929 16:03:04.814410 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.814415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.814457 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:03:04.814649 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.814654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.814752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.814871 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.815235 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.816013 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.817330 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:04.817724 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:04.817826 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:04.818933 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.818943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.818961 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.819666 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.819726 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.819747 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.819861 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.819976 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.820200 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:03:04.821261 25945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:04.918223 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25823
I20260929 16:03:04.938249 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 25471
I20260929 16:03:04.939713 25532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:05.066088 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 25471
2026-09-29T16:03:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3312 ms)
[----------] 6 tests from EncryptionPolicyTest (24121 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:05 dist-test-slave-b69v krb5kdc[26003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:05 dist-test-slave-b69v krb5kdc[26003](info): set up 2 sockets
Sep 29 16:03:05 dist-test-slave-b69v krb5kdc[26003](info): commencing operation
krb5kdc: starting...
W20260929 16:03:06.212870 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:06 dist-test-slave-b69v krb5kdc[26003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:06.539770 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:06.912847 26016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:06.913241 26016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:06.913420 26016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:06.920805 26016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:06.920956 26016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:06.921051 26016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:06.921167 26016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:06.921245 26016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:06.931200 26016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:06.932850 26016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:06.934362 26016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:06.947268 26024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:06.947333 26022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:06.947490 26021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:06.948279 26016 server_base.cc:1034] running on GCE node
I20260929 16:03:06.949136 26016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:06.950626 26016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:06.951872 26016 hybrid_clock.cc:648] HybridClock initialized: now 1790697786951836 us; error 35 us; skew 500 ppm
I20260929 16:03:06.952294 26016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:06 dist-test-slave-b69v krb5kdc[26003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:06.955694 26016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:06.957333 26016 webserver.cc:460] Webserver started at http://127.0.0.1:37035/ using document root <none> and password file <none>
I20260929 16:03:06.958094 26016 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:06.958246 26016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:06.958598 26016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:06.961973 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "0b83344f66794707ab44bcf74dcda66d"
format_stamp: "Formatted at 2026-09-29 16:03:06 on dist-test-slave-b69v"
server_key: "26a7b0ae31eaac477bf46f3b3726d006"
server_key_iv: "248b25cc404df6bc0c55b0e25b9b90ac"
server_key_version: "encryptionkey@0"
I20260929 16:03:06.962785 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "0b83344f66794707ab44bcf74dcda66d"
format_stamp: "Formatted at 2026-09-29 16:03:06 on dist-test-slave-b69v"
server_key: "26a7b0ae31eaac477bf46f3b3726d006"
server_key_iv: "248b25cc404df6bc0c55b0e25b9b90ac"
server_key_version: "encryptionkey@0"
I20260929 16:03:06.968212 26016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 16:03:06.971697 26031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:06.973089 26016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:06.973292 26016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "0b83344f66794707ab44bcf74dcda66d"
format_stamp: "Formatted at 2026-09-29 16:03:06 on dist-test-slave-b69v"
server_key: "26a7b0ae31eaac477bf46f3b3726d006"
server_key_iv: "248b25cc404df6bc0c55b0e25b9b90ac"
server_key_version: "encryptionkey@0"
I20260929 16:03:06.973497 26016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:06.999368 26016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:07.004374 26016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:07.004760 26016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:07.027856 26016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42811
I20260929 16:03:07.027920 26082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42811 every 8 connection(s)
I20260929 16:03:07.029593 26016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:07.030407 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26016
I20260929 16:03:07.030694 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:07.031591 5588 external_mini_cluster.cc:1428] Setting key 0c8d9a841bc0866d51de45111d0cfa2c
I20260929 16:03:07.035534 26083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:03:07 dist-test-slave-b69v krb5kdc[26003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:03:07.050315 26083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d: Bootstrap starting.
I20260929 16:03:07.054169 26083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:07.055804 26083 log.cc:824] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:07.059463 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:07.034411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42713 (local address 127.0.0.1:42811)
0929 16:03:07.035354 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:07.035368 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:07.035421 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:03:07.037382 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:07.037405 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:07.037823 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:07.038179 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:07.040369 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:07.042088 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:07.044183 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:07.044910 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:07.045079 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:07.050287 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:07.050299 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:07.050314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:07.050354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:07.053797 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:07.054709 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:07.054715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:07.054721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:07.054919 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:07.055417 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:07.055422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:07.055424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:07.056018 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:07.056430 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:07.057288 (+ 858us) server_negotiation.cc:299] Negotiation successful
0929 16:03:07.057872 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":439,"threads_started":1}
I20260929 16:03:07.060168 26083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d: No bootstrap required, opened a new log
I20260929 16:03:07.064532 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:07.068346 26083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260929 16:03:07.068796 26083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:07.068924 26083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b83344f66794707ab44bcf74dcda66d, State: Initialized, Role: FOLLOWER
I20260929 16:03:07.069772 26083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260929 16:03:07.070115 26083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:07.070300 26083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:07.070549 26083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:07.074061 26083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260929 16:03:07.074769 26083 leader_election.cc:302] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b83344f66794707ab44bcf74dcda66d; no voters:
I20260929 16:03:07.075580 26083 leader_election.cc:288] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:07.075714 26093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:07.079453 26083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:07.084324 26093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [term 1 LEADER]: Becoming Leader. State: Replica: 0b83344f66794707ab44bcf74dcda66d, State: Running, Role: LEADER
I20260929 16:03:07.085544 26093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } }
I20260929 16:03:07.089515 26094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b83344f66794707ab44bcf74dcda66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260929 16:03:07.089542 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b83344f66794707ab44bcf74dcda66d. Latest consensus state: current_term: 1 leader_uuid: "0b83344f66794707ab44bcf74dcda66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b83344f66794707ab44bcf74dcda66d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42811 } } }
I20260929 16:03:07.090270 26094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:07.090364 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:07.105342 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:07.070125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49855 (local address 127.0.0.1:42811)
0929 16:03:07.070544 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:07.070552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:07.072653 (+ 2101us) server_negotiation.cc:407] Connection header received
0929 16:03:07.072840 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:07.072846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:07.072929 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:07.073036 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:07.075157 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:07.077028 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:07.079015 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:07.079818 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:07.080642 (+ 824us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:07.084579 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:07.084590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:07.084596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:07.084634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:07.098039 (+ 13405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:07.099799 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:07.099808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:07.099815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:07.100086 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:07.100876 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:07.100884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:07.100890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:07.101473 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:07.104092 (+ 2619us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:07.104374 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:03:07.104684 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260929 16:03:07.110970 26107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:07.117677 26107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:07.130362 26107 catalog_manager.cc:1347] Generated new cluster ID: 03c144ffefeb48c88d66a23585a576c0
I20260929 16:03:07.130544 26107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:07.142792 26107 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:07.144357 26107 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:07.162042 26107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b83344f66794707ab44bcf74dcda66d: Generated new TSK 0
I20260929 16:03:07.163507 26107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:07.178975 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26016
I20260929 16:03:07.181176 26078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:07.283325 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26016
W20260929 16:03:07.306665 26001 connection.cc:535] client connection to 127.0.0.1:42811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42811): Connection reset by peer (error 104)
2026-09-29T16:03:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:07 dist-test-slave-b69v krb5kdc[26115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:07 dist-test-slave-b69v krb5kdc[26115](info): set up 2 sockets
Sep 29 16:03:07 dist-test-slave-b69v krb5kdc[26115](info): commencing operation
krb5kdc: starting...
W20260929 16:03:08.417083 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:08 dist-test-slave-b69v krb5kdc[26115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:08.701645 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:09.067620 26128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:09.067953 26128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:09.068159 26128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:09.075140 26128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:09.075291 26128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:09.075398 26128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:09.075518 26128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:09.075606 26128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:09.085193 26128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:09.086601 26128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:09.088037 26128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:09.100365 26133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:09.100641 26134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:09.101152 26136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:09.101847 26128 server_base.cc:1034] running on GCE node
I20260929 16:03:09.102739 26128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:09.104331 26128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:09.105610 26128 hybrid_clock.cc:648] HybridClock initialized: now 1790697789105583 us; error 60 us; skew 500 ppm
I20260929 16:03:09.105993 26128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:09 dist-test-slave-b69v krb5kdc[26115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:09.109694 26128 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:09.111677 26128 webserver.cc:460] Webserver started at http://127.0.0.1:41901/ using document root <none> and password file <none>
I20260929 16:03:09.112589 26128 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:09.112705 26128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:09.112985 26128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:09.116561 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "65a85277f945412bb3d9935132779323"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-b69v"
server_key: "3300bf6fa489392c346fea5a29f438aa"
server_key_iv: "a9c8463256de7d6598bc050bf99c5847"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.117419 26128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "65a85277f945412bb3d9935132779323"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-b69v"
server_key: "3300bf6fa489392c346fea5a29f438aa"
server_key_iv: "a9c8463256de7d6598bc050bf99c5847"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.123972 26128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 16:03:09.128288 26143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:09.129904 26128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:09.130093 26128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "65a85277f945412bb3d9935132779323"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-b69v"
server_key: "3300bf6fa489392c346fea5a29f438aa"
server_key_iv: "a9c8463256de7d6598bc050bf99c5847"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.130306 26128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:09.164247 26128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:09.169289 26128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:09.169569 26128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:09.191305 26128 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36371
I20260929 16:03:09.191350 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36371 every 8 connection(s)
I20260929 16:03:09.192916 26128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:09.197278 26195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:09.197423 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26128
I20260929 16:03:09.197633 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:09.198284 5588 external_mini_cluster.cc:1428] Setting key 192a95458ea313061e45c07003de1280
I20260929 16:03:09.212812 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:09.200746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60997 (local address 127.0.0.1:36371)
0929 16:03:09.201685 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:09.201697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:09.201746 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:03:09.203782 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:09.203808 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:09.204320 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:09.204759 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:09.205349 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:09.206757 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:09.208224 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:09.208919 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:09.209124 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:09.210420 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:09.210429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:09.210446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:09.211033 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:09.211117 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:09.211146 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:09.211315 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:09.211560 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:03:09.211729 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":425,"threads_started":1}
I20260929 16:03:09.213114 26195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323: Bootstrap starting.
I20260929 16:03:09.216779 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:09.216987 26195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:09.217980 26195 log.cc:824] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:09.220883 26195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323: No bootstrap required, opened a new log
I20260929 16:03:09.228407 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:09.221699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51545 (local address 127.0.0.1:36371)
0929 16:03:09.221911 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:09.221919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:09.222227 (+ 308us) server_negotiation.cc:407] Connection header received
0929 16:03:09.222398 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:09.222401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:09.222457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:09.222531 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:09.223143 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:09.223927 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:09.225790 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:09.226271 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:09.226371 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:09.227576 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:09.227581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:09.227584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:09.227600 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:09.227637 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:09.227651 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:09.227737 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:09.227936 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 16:03:09.227974 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:03:09.231194 26195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260929 16:03:09.231566 26195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:09.231690 26195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a85277f945412bb3d9935132779323, State: Initialized, Role: FOLLOWER
I20260929 16:03:09.232331 26195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260929 16:03:09.232606 26195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:09.232775 26195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:09.232949 26195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:09.235760 26195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260929 16:03:09.236373 26195 leader_election.cc:302] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a85277f945412bb3d9935132779323; no voters:
I20260929 16:03:09.236969 26195 leader_election.cc:288] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:09.237128 26205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:09.237854 26205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [term 1 LEADER]: Becoming Leader. State: Replica: 65a85277f945412bb3d9935132779323, State: Running, Role: LEADER
I20260929 16:03:09.238516 26205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } }
I20260929 16:03:09.239180 26195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:09.241398 26207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a85277f945412bb3d9935132779323. Latest consensus state: current_term: 1 leader_uuid: "65a85277f945412bb3d9935132779323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } } }
I20260929 16:03:09.241846 26207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:09.241712 26206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a85277f945412bb3d9935132779323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a85277f945412bb3d9935132779323" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36371 } } }
I20260929 16:03:09.241993 26206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:09.243892 26211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:09.249454 26211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:09.259541 26211 catalog_manager.cc:1347] Generated new cluster ID: 6135540b877e47568c73adfa926c64eb
I20260929 16:03:09.259675 26211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:09.299844 26211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:09.301483 26211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:09.311689 26211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65a85277f945412bb3d9935132779323: Generated new TSK 0
I20260929 16:03:09.313068 26211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:09.333705 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26128
W20260929 16:03:09.335287 26190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60997
W20260929 16:03:09.336381 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:09.336591 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26128
W20260929 16:03:09.358728 26113 connection.cc:535] client connection to 127.0.0.1:36371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36371): Connection reset by peer (error 104)
2026-09-29T16:03:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:09 dist-test-slave-b69v krb5kdc[26226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:09 dist-test-slave-b69v krb5kdc[26226](info): set up 2 sockets
Sep 29 16:03:09 dist-test-slave-b69v krb5kdc[26226](info): commencing operation
krb5kdc: starting...
W20260929 16:03:10.448951 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:10 dist-test-slave-b69v krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:10.735666 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:11.117961 26239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:11.118389 26239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:11.118578 26239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:11.126027 26239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:11.126176 26239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:11.126283 26239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:11.126415 26239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:11.126503 26239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:11.136546 26239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:11.137995 26239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:11.139547 26239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:11.152925 26244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:11.152882 26245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:11.153515 26247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:11.154100 26239 server_base.cc:1034] running on GCE node
I20260929 16:03:11.154958 26239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:11.156654 26239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:11.157888 26239 hybrid_clock.cc:648] HybridClock initialized: now 1790697791157826 us; error 49 us; skew 500 ppm
I20260929 16:03:11.158257 26239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:11 dist-test-slave-b69v krb5kdc[26226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:11.162050 26239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:11.163794 26239 webserver.cc:460] Webserver started at http://127.0.0.1:33769/ using document root <none> and password file <none>
I20260929 16:03:11.164605 26239 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:11.164748 26239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:11.165098 26239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:11.168319 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "257a29eaec9e4dc281bc67803501f7b2"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-b69v"
server_key: "002a313f9b95ba8e6593120f0194246b"
server_key_iv: "baa8bd0e0d21c58f4c28d08e16a2864d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.169133 26239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "257a29eaec9e4dc281bc67803501f7b2"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-b69v"
server_key: "002a313f9b95ba8e6593120f0194246b"
server_key_iv: "baa8bd0e0d21c58f4c28d08e16a2864d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.175637 26239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:03:11.179241 26254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:11.180632 26239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:11.180835 26239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "257a29eaec9e4dc281bc67803501f7b2"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-b69v"
server_key: "002a313f9b95ba8e6593120f0194246b"
server_key_iv: "baa8bd0e0d21c58f4c28d08e16a2864d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.181039 26239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:11.186932 26239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:11.191380 26239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:11.191690 26239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:11.210810 26239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42945
I20260929 16:03:11.210862 26305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42945 every 8 connection(s)
I20260929 16:03:11.212198 26239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:11.214340 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26239
I20260929 16:03:11.214522 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:11.215000 5588 external_mini_cluster.cc:1428] Setting key 2a001b15b1bf90a44fb938252bbe0e41
I20260929 16:03:11.216499 26306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:11.222532 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:11.216827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42519 (local address 127.0.0.1:42945)
0929 16:03:11.217569 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:11.217581 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:11.217637 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:03:11.219241 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:11.219261 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:11.219592 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:11.219969 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:11.221012 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:11.221018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:11.221029 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:11.221043 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:11.221128 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:11.221150 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:11.221317 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:11.221422 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:03:11.221622 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":350,"threads_started":1}
I20260929 16:03:11.226120 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:11.229831 26306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: Bootstrap starting.
I20260929 16:03:11.234198 26306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:11.234392 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:11.231813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41255 (local address 127.0.0.1:42945)
0929 16:03:11.232017 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:11.232023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:11.232328 (+ 305us) server_negotiation.cc:407] Connection header received
0929 16:03:11.232491 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:11.232494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:11.232538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:11.232613 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:11.233527 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:11.233530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:11.233533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:11.233559 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:11.233592 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:11.233605 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:11.233687 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:11.233835 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:03:11.233875 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:03:11.235467 26306 log.cc:824] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:11.238097 26306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: No bootstrap required, opened a new log
I20260929 16:03:11.245034 26306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } }
I20260929 16:03:11.245393 26306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:11.245522 26306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257a29eaec9e4dc281bc67803501f7b2, State: Initialized, Role: FOLLOWER
I20260929 16:03:11.246093 26306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } }
I20260929 16:03:11.246349 26306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:11.246510 26306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:11.246690 26306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:11.248837 26306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } }
I20260929 16:03:11.249332 26306 leader_election.cc:302] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257a29eaec9e4dc281bc67803501f7b2; no voters:
I20260929 16:03:11.249917 26306 leader_election.cc:288] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:11.250017 26316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:11.250839 26316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [term 1 LEADER]: Becoming Leader. State: Replica: 257a29eaec9e4dc281bc67803501f7b2, State: Running, Role: LEADER
I20260929 16:03:11.251426 26316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } }
I20260929 16:03:11.252022 26306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:11.254451 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257a29eaec9e4dc281bc67803501f7b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } } }
I20260929 16:03:11.255088 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:11.254465 26318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257a29eaec9e4dc281bc67803501f7b2. Latest consensus state: current_term: 1 leader_uuid: "257a29eaec9e4dc281bc67803501f7b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a29eaec9e4dc281bc67803501f7b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42945 } } }
I20260929 16:03:11.255625 26318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2 [sys.catalog]: This master's current role is: LEADER
W20260929 16:03:11.264019 26332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:11.264197 26332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:11.264348 26324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:11.266644 26324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:11.275070 26324 catalog_manager.cc:1347] Generated new cluster ID: db9caee068554e33a2f5c95e4a96c150
I20260929 16:03:11.275203 26324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:11.285665 26324 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:11.286063 26324 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:11.296720 26324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257a29eaec9e4dc281bc67803501f7b2: Generated new TSK 0
I20260929 16:03:11.297588 26324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:11.313297 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26239
W20260929 16:03:11.314553 26301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42519
W20260929 16:03:11.315281 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:11.315438 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26239
2026-09-29T16:03:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:11 dist-test-slave-b69v krb5kdc[26337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:11 dist-test-slave-b69v krb5kdc[26337](info): set up 2 sockets
Sep 29 16:03:11 dist-test-slave-b69v krb5kdc[26337](info): commencing operation
krb5kdc: starting...
W20260929 16:03:12.444720 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.009s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:12 dist-test-slave-b69v krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:12.706981 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:13.066220 26350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:13.066596 26350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:13.066784 26350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:13.074052 26350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:13.074218 26350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:13.074354 26350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:13.074470 26350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:13.074558 26350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:13.084127 26350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:13.085505 26350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:13.086920 26350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:13.100834 26358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:13.101048 26356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:13.101066 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:13.102061 26350 server_base.cc:1034] running on GCE node
I20260929 16:03:13.102926 26350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:13.104784 26350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:13.106060 26350 hybrid_clock.cc:648] HybridClock initialized: now 1790697793105992 us; error 81 us; skew 500 ppm
I20260929 16:03:13.106465 26350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:13 dist-test-slave-b69v krb5kdc[26337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:13.110397 26350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:13.112298 26350 webserver.cc:460] Webserver started at http://127.0.0.1:38099/ using document root <none> and password file <none>
I20260929 16:03:13.113577 26350 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:13.113763 26350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:13.114135 26350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:13.117700 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "f5e3964fd38b4b9bb7538872fb338837"
format_stamp: "Formatted at 2026-09-29 16:03:13 on dist-test-slave-b69v"
server_key: "876fbb4b208e4ace3a6f0a7d01b8be95"
server_key_iv: "78b18b171e5e9dd26b08c6e3d337e618"
server_key_version: "encryptionkey@0"
I20260929 16:03:13.118561 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "f5e3964fd38b4b9bb7538872fb338837"
format_stamp: "Formatted at 2026-09-29 16:03:13 on dist-test-slave-b69v"
server_key: "876fbb4b208e4ace3a6f0a7d01b8be95"
server_key_iv: "78b18b171e5e9dd26b08c6e3d337e618"
server_key_version: "encryptionkey@0"
I20260929 16:03:13.124356 26350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 16:03:13.128247 26365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:13.129694 26350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:13.129899 26350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "f5e3964fd38b4b9bb7538872fb338837"
format_stamp: "Formatted at 2026-09-29 16:03:13 on dist-test-slave-b69v"
server_key: "876fbb4b208e4ace3a6f0a7d01b8be95"
server_key_iv: "78b18b171e5e9dd26b08c6e3d337e618"
server_key_version: "encryptionkey@0"
I20260929 16:03:13.130116 26350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:13.156104 26350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:13.160914 26350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:13.161237 26350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:13.183619 26350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44955
I20260929 16:03:13.183645 26416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44955 every 8 connection(s)
I20260929 16:03:13.185302 26350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:13.189282 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26350
I20260929 16:03:13.189495 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:13.189811 26417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:13.190215 5588 external_mini_cluster.cc:1428] Setting key ad4591610aa460e4104520572b9294bf
I20260929 16:03:13.202544 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837: Bootstrap starting.
Sep 29 16:03:13 dist-test-slave-b69v krb5kdc[26337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:03:13.206162 26417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:13.207355 26417 log.cc:824] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:13.210619 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837: No bootstrap required, opened a new log
I20260929 16:03:13.212579 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:13.192103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57065 (local address 127.0.0.1:44955)
0929 16:03:13.192931 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:13.192944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:13.192989 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:03:13.194987 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:13.195009 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:13.195392 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:13.195775 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:13.197256 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:13.198477 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:13.199769 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:13.200582 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:13.200822 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:13.205188 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:13.205195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:13.205207 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:13.205254 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:13.208590 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:13.209470 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:13.209476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:13.209481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:13.209643 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:13.210074 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:13.210077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:13.210079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:13.210519 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:13.210799 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:13.211175 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 16:03:13.211514 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":402,"threads_started":1}
I20260929 16:03:13.216091 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:13.219194 26417 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260929 16:03:13.219676 26417 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:13.219832 26417 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e3964fd38b4b9bb7538872fb338837, State: Initialized, Role: FOLLOWER
I20260929 16:03:13.220572 26417 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260929 16:03:13.220855 26417 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:13.221002 26417 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:13.221177 26417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:13.224298 26417 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260929 16:03:13.224839 26417 leader_election.cc:302] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5e3964fd38b4b9bb7538872fb338837; no voters:
I20260929 16:03:13.225505 26417 leader_election.cc:288] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:13.225605 26427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:13.226822 26427 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [term 1 LEADER]: Becoming Leader. State: Replica: f5e3964fd38b4b9bb7538872fb338837, State: Running, Role: LEADER
I20260929 16:03:13.227411 26427 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } }
I20260929 16:03:13.227600 26417 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:13.236466 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5e3964fd38b4b9bb7538872fb338837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260929 16:03:13.236900 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:13.237391 26429 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5e3964fd38b4b9bb7538872fb338837. Latest consensus state: current_term: 1 leader_uuid: "f5e3964fd38b4b9bb7538872fb338837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5e3964fd38b4b9bb7538872fb338837" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44955 } } }
I20260929 16:03:13.237769 26429 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:13.238880 26438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:13.239974 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:13.221604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40729 (local address 127.0.0.1:44955)
0929 16:03:13.221807 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:13.221813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:13.222149 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:03:13.222307 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:13.222310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:13.222357 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:13.222438 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:13.223986 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:13.224702 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:13.226431 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:13.227124 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:13.227253 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:13.230582 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:13.230591 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:13.230599 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:13.230648 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:13.235785 (+ 5137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:13.236669 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:13.236672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:13.236678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:13.236791 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:13.237297 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:13.237304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:13.237310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:13.237823 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:13.239213 (+ 1390us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:13.239359 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:03:13.239569 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:03:13.245635 26438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:13.256150 26438 catalog_manager.cc:1347] Generated new cluster ID: b99abd47ebe94f209910a89369628554
I20260929 16:03:13.256266 26438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:13.269845 26438 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:13.271761 26438 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:13.279057 26438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5e3964fd38b4b9bb7538872fb338837: Generated new TSK 0
I20260929 16:03:13.279878 26438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:13.285339 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26350
I20260929 16:03:13.286792 26412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:13.380146 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26350
2026-09-29T16:03:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:13 dist-test-slave-b69v krb5kdc[26449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:13 dist-test-slave-b69v krb5kdc[26449](info): set up 2 sockets
Sep 29 16:03:13 dist-test-slave-b69v krb5kdc[26449](info): commencing operation
krb5kdc: starting...
W20260929 16:03:14.504917 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:14 dist-test-slave-b69v krb5kdc[26449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:14.795428 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:15.156629 26462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:15.156965 26462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:15.157158 26462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:15.164281 26462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:15.164412 26462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:15.164516 26462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:15.164628 26462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:15.164703 26462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:15.174196 26462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:15.175736 26462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:15.177223 26462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:15.189323 26470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:15.189419 26467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:15.189424 26468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:15.190608 26462 server_base.cc:1034] running on GCE node
I20260929 16:03:15.191378 26462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:15.192823 26462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:15.194056 26462 hybrid_clock.cc:648] HybridClock initialized: now 1790697795194011 us; error 60 us; skew 500 ppm
I20260929 16:03:15.194404 26462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:15 dist-test-slave-b69v krb5kdc[26449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:15.201491 26462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:15.202952 26462 webserver.cc:460] Webserver started at http://127.0.0.1:36357/ using document root <none> and password file <none>
I20260929 16:03:15.203615 26462 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:15.203758 26462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:15.204102 26462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:15.206866 26462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "9f2c1f151cb547c4b7d090fe414014fc"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-b69v"
server_key: "72f70a855e848939d67b534e507981f6"
server_key_iv: "019f4edce8e21f4b1e4ffa8cf68a6446"
server_key_version: "encryptionkey@0"
I20260929 16:03:15.207726 26462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "9f2c1f151cb547c4b7d090fe414014fc"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-b69v"
server_key: "72f70a855e848939d67b534e507981f6"
server_key_iv: "019f4edce8e21f4b1e4ffa8cf68a6446"
server_key_version: "encryptionkey@0"
I20260929 16:03:15.212773 26462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:03:15.216028 26477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:15.217264 26462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 16:03:15.217447 26462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "9f2c1f151cb547c4b7d090fe414014fc"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-b69v"
server_key: "72f70a855e848939d67b534e507981f6"
server_key_iv: "019f4edce8e21f4b1e4ffa8cf68a6446"
server_key_version: "encryptionkey@0"
I20260929 16:03:15.217625 26462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:15.242283 26462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:15.246331 26462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:15.246613 26462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:15.267468 26462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33503
I20260929 16:03:15.267506 26528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33503 every 8 connection(s)
I20260929 16:03:15.269240 26462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:15.274232 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:15.277017 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26462
I20260929 16:03:15.277275 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:15.277885 5588 external_mini_cluster.cc:1428] Setting key 58dd20af74aea313fc5179647a53abdc
I20260929 16:03:15.287681 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc: Bootstrap starting.
I20260929 16:03:15.291270 26529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:15.292590 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:15.280220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60453 (local address 127.0.0.1:33503)
0929 16:03:15.281134 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:15.281148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:15.281197 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:03:15.283077 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:15.283101 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:15.283535 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:15.283967 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:15.284621 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:15.286123 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:15.287565 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:15.288432 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:15.288696 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:15.289988 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:15.289996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:15.290010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:15.290595 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:15.290674 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:15.290701 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:15.290841 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:15.291072 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:03:15.291341 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":413,"threads_started":1}
I20260929 16:03:15.292595 26529 log.cc:824] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:15.295781 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc: No bootstrap required, opened a new log
I20260929 16:03:15.296484 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:15.304538 26529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } }
I20260929 16:03:15.304919 26529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:15.305019 26529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2c1f151cb547c4b7d090fe414014fc, State: Initialized, Role: FOLLOWER
I20260929 16:03:15.305716 26529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } }
I20260929 16:03:15.306044 26529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:15.306197 26529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:15.306421 26529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:15.308699 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:15.301152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34723 (local address 127.0.0.1:33503)
0929 16:03:15.301370 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:15.301376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:15.301907 (+ 531us) server_negotiation.cc:407] Connection header received
0929 16:03:15.302048 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:15.302053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:15.302132 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:15.302248 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:15.302732 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:15.303842 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:15.305448 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:15.306168 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:15.306325 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:15.307808 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:15.307814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:15.307819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:15.307839 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:15.307881 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:15.307902 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:15.308010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:15.308292 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 16:03:15.308385 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:03:15.309207 26529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } }
I20260929 16:03:15.309728 26529 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2c1f151cb547c4b7d090fe414014fc; no voters:
I20260929 16:03:15.310384 26529 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:15.310647 26539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:15.311770 26539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [term 1 LEADER]: Becoming Leader. State: Replica: 9f2c1f151cb547c4b7d090fe414014fc, State: Running, Role: LEADER
I20260929 16:03:15.312599 26539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } }
I20260929 16:03:15.313123 26529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:15.317143 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2c1f151cb547c4b7d090fe414014fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } } }
I20260929 16:03:15.317616 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:15.318212 26541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2c1f151cb547c4b7d090fe414014fc. Latest consensus state: current_term: 1 leader_uuid: "9f2c1f151cb547c4b7d090fe414014fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2c1f151cb547c4b7d090fe414014fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33503 } } }
I20260929 16:03:15.318656 26541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:15.320788 26548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:15.328704 26548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:15.338848 26548 catalog_manager.cc:1347] Generated new cluster ID: c958f7bb1b2d4bfb91c6326b50f24ee7
I20260929 16:03:15.338994 26548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:15.349956 26548 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:15.351765 26548 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:15.358377 26548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2c1f151cb547c4b7d090fe414014fc: Generated new TSK 0
I20260929 16:03:15.359282 26548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:15.377112 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26462
W20260929 16:03:15.378082 26524 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60453
W20260929 16:03:15.378741 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26462: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:15.378886 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26462
2026-09-29T16:03:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:15 dist-test-slave-b69v krb5kdc[26561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:15 dist-test-slave-b69v krb5kdc[26561](info): set up 2 sockets
Sep 29 16:03:15 dist-test-slave-b69v krb5kdc[26561](info): commencing operation
krb5kdc: starting...
W20260929 16:03:16.473402 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:16 dist-test-slave-b69v krb5kdc[26561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:16.742689 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:17.108554 26574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:17.108889 26574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:17.109035 26574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:17.116245 26574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:17.116397 26574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:17.116490 26574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:17.116601 26574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:17.116688 26574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:17.128743 26574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:17.130321 26574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:17.131733 26574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:17.146261 26579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:17.146240 26582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:17.146314 26580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:17.147172 26574 server_base.cc:1034] running on GCE node
I20260929 16:03:17.148490 26574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:17.150193 26574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:17.151463 26574 hybrid_clock.cc:648] HybridClock initialized: now 1790697797151364 us; error 112 us; skew 500 ppm
I20260929 16:03:17.152141 26574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:17 dist-test-slave-b69v krb5kdc[26561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:17.155673 26574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:17.157330 26574 webserver.cc:460] Webserver started at http://127.0.0.1:41275/ using document root <none> and password file <none>
I20260929 16:03:17.158113 26574 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:17.158267 26574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:17.158614 26574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:17.161950 26574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "00a0df0a22d049168f735e73a064d4ca"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-b69v"
server_key: "b49cff32655f820b6c195650e818d085"
server_key_iv: "b62ae94cd9ae386b877c26caf2473fd3"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.162767 26574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "00a0df0a22d049168f735e73a064d4ca"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-b69v"
server_key: "b49cff32655f820b6c195650e818d085"
server_key_iv: "b62ae94cd9ae386b877c26caf2473fd3"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.168838 26574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:03:17.173110 26589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:17.174561 26574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:03:17.174773 26574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "00a0df0a22d049168f735e73a064d4ca"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-b69v"
server_key: "b49cff32655f820b6c195650e818d085"
server_key_iv: "b62ae94cd9ae386b877c26caf2473fd3"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.174990 26574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:17.181218 26574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:17.185813 26574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:17.186123 26574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:17.207101 26574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34421
I20260929 16:03:17.207180 26640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34421 every 8 connection(s)
I20260929 16:03:17.208509 26574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:17.212553 26641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:17.216240 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26574
I20260929 16:03:17.216419 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:17.216871 5588 external_mini_cluster.cc:1428] Setting key 9eb6d5184f75a82146337c7ac232faaf
I20260929 16:03:17.224601 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:17.218614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.0.0.1:34421)
0929 16:03:17.219498 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:17.219510 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:17.219553 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:03:17.221300 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:17.221321 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:17.221664 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:17.221986 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:17.223034 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:17.223039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:17.223053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:17.223066 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:17.223122 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:17.223142 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:17.223314 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:17.223410 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:03:17.223608 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":431,"threads_started":1}
I20260929 16:03:17.225219 26641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca: Bootstrap starting.
I20260929 16:03:17.228017 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:17.228585 26641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:17.229660 26641 log.cc:824] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:17.233170 26641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca: No bootstrap required, opened a new log
I20260929 16:03:17.236284 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:17.233073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46989 (local address 127.0.0.1:34421)
0929 16:03:17.233320 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:17.233331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:17.233908 (+ 577us) server_negotiation.cc:407] Connection header received
0929 16:03:17.234148 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:17.234153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:17.234208 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:17.234324 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:17.235409 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:17.235415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:17.235420 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:17.235444 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:17.235493 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:17.235512 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:17.235645 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:17.235785 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:03:17.235864 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:03:17.242231 26641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260929 16:03:17.242715 26641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:17.242827 26641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a0df0a22d049168f735e73a064d4ca, State: Initialized, Role: FOLLOWER
I20260929 16:03:17.243515 26641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260929 16:03:17.243845 26641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:17.244029 26641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:17.244298 26641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:17.246622 26641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260929 16:03:17.247113 26641 leader_election.cc:302] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a0df0a22d049168f735e73a064d4ca; no voters:
I20260929 16:03:17.247745 26641 leader_election.cc:288] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:17.247978 26651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:17.248787 26651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [term 1 LEADER]: Becoming Leader. State: Replica: 00a0df0a22d049168f735e73a064d4ca, State: Running, Role: LEADER
I20260929 16:03:17.249310 26651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } }
I20260929 16:03:17.250078 26641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:17.253549 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a0df0a22d049168f735e73a064d4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } } }
I20260929 16:03:17.254069 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:17.254674 26653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a0df0a22d049168f735e73a064d4ca. Latest consensus state: current_term: 1 leader_uuid: "00a0df0a22d049168f735e73a064d4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a0df0a22d049168f735e73a064d4ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34421 } } }
I20260929 16:03:17.255107 26653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:17.256755 26657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:17.262880 26657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:17.272889 26657 catalog_manager.cc:1347] Generated new cluster ID: 125c4a92ff3e4180b072d7b7bcdb9b64
I20260929 16:03:17.273038 26657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:17.296314 26657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:17.296728 26657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:17.309314 26657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a0df0a22d049168f735e73a064d4ca: Generated new TSK 0
I20260929 16:03:17.310223 26657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:17.316694 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26574
W20260929 16:03:17.317961 26636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37089
W20260929 16:03:17.318660 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:17.318812 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26574
2026-09-29T16:03:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:17 dist-test-slave-b69v krb5kdc[26672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:17 dist-test-slave-b69v krb5kdc[26672](info): set up 2 sockets
Sep 29 16:03:17 dist-test-slave-b69v krb5kdc[26672](info): commencing operation
krb5kdc: starting...
W20260929 16:03:18.442457 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:18 dist-test-slave-b69v krb5kdc[26672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:18.740720 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:19.127128 26685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:19.127445 26685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:19.127611 26685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:19.134708 26685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:19.134809 26685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:19.134927 26685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:19.135064 26685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:19.135149 26685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:19.144666 26685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:19.145996 26685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:19.147394 26685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:19.160578 26690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:19.160612 26693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:19.160640 26691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:19.161748 26685 server_base.cc:1034] running on GCE node
I20260929 16:03:19.162552 26685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:19.163990 26685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:19.165292 26685 hybrid_clock.cc:648] HybridClock initialized: now 1790697799165248 us; error 38 us; skew 500 ppm
I20260929 16:03:19.165710 26685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:19 dist-test-slave-b69v krb5kdc[26672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:19.169134 26685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:19.170644 26685 webserver.cc:460] Webserver started at http://127.0.0.1:33699/ using document root <none> and password file <none>
I20260929 16:03:19.171381 26685 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:19.171535 26685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:19.171855 26685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:19.175064 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "7998a34d44a04f68bb756c06ef9516e4"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-b69v"
server_key: "be28cb13b87de4675b92e0ee1097eb81"
server_key_iv: "b7bf78dae5a2dd4241b6a9bbf60ba03a"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.175863 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "7998a34d44a04f68bb756c06ef9516e4"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-b69v"
server_key: "be28cb13b87de4675b92e0ee1097eb81"
server_key_iv: "b7bf78dae5a2dd4241b6a9bbf60ba03a"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.181279 26685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 16:03:19.184661 26700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:19.185897 26685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:19.186086 26685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "7998a34d44a04f68bb756c06ef9516e4"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-b69v"
server_key: "be28cb13b87de4675b92e0ee1097eb81"
server_key_iv: "b7bf78dae5a2dd4241b6a9bbf60ba03a"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.186273 26685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:19.199137 26685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:19.203006 26685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:19.203287 26685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:19.222260 26685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46795
I20260929 16:03:19.222293 26751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46795 every 8 connection(s)
I20260929 16:03:19.223508 26685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:19.227263 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:19.228509 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26685
I20260929 16:03:19.228719 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:19.229310 5588 external_mini_cluster.cc:1428] Setting key 9402e1399257ce4d71b8cac43abdc1ab
I20260929 16:03:19.242650 26752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4: Bootstrap starting.
Sep 29 16:03:19 dist-test-slave-b69v krb5kdc[26672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:03:19.246876 26752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:19.248557 26752 log.cc:824] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:19.252174 26752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4: No bootstrap required, opened a new log
I20260929 16:03:19.254904 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.231249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:51211 (local address 127.0.0.1:46795)
0929 16:03:19.232136 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.232148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.232193 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:03:19.233984 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.234008 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.234429 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.234790 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.236805 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.238151 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.240011 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.240773 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.240906 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.246170 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.246177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.246194 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.246233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.250186 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.250963 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.250977 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.250985 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.251205 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.251642 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.251646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.251649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.252407 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.252775 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.253100 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.253477 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260929 16:03:19.259188 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:19.261680 26752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260929 16:03:19.262064 26752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:19.262194 26752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7998a34d44a04f68bb756c06ef9516e4, State: Initialized, Role: FOLLOWER
I20260929 16:03:19.262835 26752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260929 16:03:19.263103 26752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:19.263254 26752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:19.263432 26752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:19.266156 26752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260929 16:03:19.266714 26752 leader_election.cc:302] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7998a34d44a04f68bb756c06ef9516e4; no voters:
I20260929 16:03:19.267467 26752 leader_election.cc:288] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:19.267681 26762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:19.269042 26762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7998a34d44a04f68bb756c06ef9516e4, State: Running, Role: LEADER
I20260929 16:03:19.269692 26762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260929 16:03:19.270110 26752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:19.272758 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7998a34d44a04f68bb756c06ef9516e4. Latest consensus state: current_term: 1 leader_uuid: "7998a34d44a04f68bb756c06ef9516e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260929 16:03:19.273239 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:19.273943 26763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7998a34d44a04f68bb756c06ef9516e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7998a34d44a04f68bb756c06ef9516e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260929 16:03:19.274329 26763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:19.277541 26767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:19.282869 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.264192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:53977 (local address 127.0.0.1:46795)
0929 16:03:19.264382 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.264388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.264847 (+ 459us) server_negotiation.cc:407] Connection header received
0929 16:03:19.264953 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.264963 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.265042 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.265160 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.266670 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.267916 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.269699 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.270246 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.270392 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.273813 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.273822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.273830 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.273880 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.278356 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.279448 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.279453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.279456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.279608 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.280285 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.280293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.280296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.280718 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.280997 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.282285 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.282494 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:03:19.284008 26767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:19.298036 26767 catalog_manager.cc:1347] Generated new cluster ID: ba7331da802c4ae7a4296639727cb254
I20260929 16:03:19.298146 26767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:19.311965 26767 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:19.313894 26767 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:19.331410 26767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7998a34d44a04f68bb756c06ef9516e4: Generated new TSK 0
I20260929 16:03:19.332482 26767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:19.348661 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26685
I20260929 16:03:19.350111 26747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:19.447657 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26685
W20260929 16:03:19.470475 26670 connection.cc:535] client connection to 127.0.0.1:46795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46795): Connection reset by peer (error 104)
2026-09-29T16:03:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2133 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:19 dist-test-slave-b69v krb5kdc[26785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:19 dist-test-slave-b69v krb5kdc[26785](info): set up 2 sockets
Sep 29 16:03:19 dist-test-slave-b69v krb5kdc[26785](info): commencing operation
krb5kdc: starting...
W20260929 16:03:20.573052 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:20 dist-test-slave-b69v krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:20.848237 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:21.236483 26798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:21.236824 26798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:21.236931 26798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:21.244733 26798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:21.244891 26798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:21.244943 26798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:21.245004 26798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:21.245052 26798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:21.255983 26798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:21.257668 26798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:21.259164 26798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:21.273100 26806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:21.273218 26804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:21.273283 26803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:21.274829 26798 server_base.cc:1034] running on GCE node
I20260929 16:03:21.275738 26798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:21.277545 26798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:21.278802 26798 hybrid_clock.cc:648] HybridClock initialized: now 1790697801278750 us; error 52 us; skew 500 ppm
I20260929 16:03:21.279202 26798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:21 dist-test-slave-b69v krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:21.290870 26798 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:21.292869 26798 webserver.cc:460] Webserver started at http://127.0.0.1:37333/ using document root <none> and password file <none>
I20260929 16:03:21.293742 26798 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:21.293905 26798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:21.294281 26798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:21.298000 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "7304cce8a2694b17927ed6c99cbb3ad8"
format_stamp: "Formatted at 2026-09-29 16:03:21 on dist-test-slave-b69v"
server_key: "27d48f2d1a266e14f1343b4200f4aaa6"
server_key_iv: "56e1f75c628ee7e1a00d93660f4ea830"
server_key_version: "encryptionkey@0"
I20260929 16:03:21.298954 26798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "7304cce8a2694b17927ed6c99cbb3ad8"
format_stamp: "Formatted at 2026-09-29 16:03:21 on dist-test-slave-b69v"
server_key: "27d48f2d1a266e14f1343b4200f4aaa6"
server_key_iv: "56e1f75c628ee7e1a00d93660f4ea830"
server_key_version: "encryptionkey@0"
I20260929 16:03:21.305472 26798 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:03:21.309500 26813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:21.311069 26798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:21.311277 26798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "7304cce8a2694b17927ed6c99cbb3ad8"
format_stamp: "Formatted at 2026-09-29 16:03:21 on dist-test-slave-b69v"
server_key: "27d48f2d1a266e14f1343b4200f4aaa6"
server_key_iv: "56e1f75c628ee7e1a00d93660f4ea830"
server_key_version: "encryptionkey@0"
I20260929 16:03:21.311510 26798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:21.335632 26798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:21.339955 26798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:21.340323 26798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:21.361832 26798 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39113
I20260929 16:03:21.361867 26864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39113 every 8 connection(s)
I20260929 16:03:21.363535 26798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:21.368090 26865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:21.372782 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26798
I20260929 16:03:21.373016 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:21.373592 5588 external_mini_cluster.cc:1428] Setting key 0dfea507300c443edb1e11682ade808c
I20260929 16:03:21.380194 26865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8: Bootstrap starting.
I20260929 16:03:21.384307 26865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:21.385820 26865 log.cc:824] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:21.388908 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:21.375621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:34063 (local address 127.0.0.1:39113)
0929 16:03:21.376510 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:21.376523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:21.376571 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:03:21.378492 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:21.378516 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:21.378921 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:21.379256 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:21.379931 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:21.381405 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:21.383374 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:21.384032 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:21.384252 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:21.385782 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:21.385790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:21.385808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:21.386715 (+ 907us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:21.386839 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:21.386879 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:21.387076 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:21.387336 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:03:21.387531 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":417,"threads_started":1}
I20260929 16:03:21.389636 26865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8: No bootstrap required, opened a new log
I20260929 16:03:21.393512 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:21.400307 26865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260929 16:03:21.401077 26865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:21.401219 26865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7304cce8a2694b17927ed6c99cbb3ad8, State: Initialized, Role: FOLLOWER
I20260929 16:03:21.402087 26865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260929 16:03:21.402462 26865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:21.402627 26865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:21.402824 26865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:21.406078 26865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260929 16:03:21.406863 26865 leader_election.cc:302] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7304cce8a2694b17927ed6c99cbb3ad8; no voters:
I20260929 16:03:21.407836 26865 leader_election.cc:288] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:21.408154 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:21.398693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:35197 (local address 127.0.0.1:39113)
0929 16:03:21.398979 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:21.398986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:21.400812 (+ 1826us) server_negotiation.cc:407] Connection header received
0929 16:03:21.400907 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:21.400914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:21.401010 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:21.401137 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:21.401751 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:21.402997 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:21.404501 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:21.405247 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:21.405384 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:21.406630 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:21.406636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:21.406641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:21.406664 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:21.406736 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:21.406759 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:21.406895 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:21.407455 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 16:03:21.407564 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 16:03:21.408178 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:21.409029 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 7304cce8a2694b17927ed6c99cbb3ad8, State: Running, Role: LEADER
I20260929 16:03:21.409747 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } }
I20260929 16:03:21.412842 26865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:21.413244 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260929 16:03:21.413272 26876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7304cce8a2694b17927ed6c99cbb3ad8. Latest consensus state: current_term: 1 leader_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7304cce8a2694b17927ed6c99cbb3ad8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39113 } } }
I20260929 16:03:21.413728 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:21.413729 26876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:21.419561 26879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:21.425724 26879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:21.437134 26879 catalog_manager.cc:1347] Generated new cluster ID: 98304d67d02749da897967915d9b9744
I20260929 16:03:21.437309 26879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:21.456954 26879 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:21.459170 26879 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:21.469308 26879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7304cce8a2694b17927ed6c99cbb3ad8: Generated new TSK 0
I20260929 16:03:21.470258 26879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:21.481856 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26798
W20260929 16:03:21.483134 26860 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.133:34063
W20260929 16:03:21.483966 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26798: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:21.484131 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26798
W20260929 16:03:21.502746 26783 connection.cc:535] client connection to 127.0.0.1:39113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39113): Connection reset by peer (error 104)
2026-09-29T16:03:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:21 dist-test-slave-b69v krb5kdc[26896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:21 dist-test-slave-b69v krb5kdc[26896](info): set up 2 sockets
Sep 29 16:03:21 dist-test-slave-b69v krb5kdc[26896](info): commencing operation
krb5kdc: starting...
W20260929 16:03:22.624873 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:22 dist-test-slave-b69v krb5kdc[26896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:22.979984 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:23.363776 26909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:23.364213 26909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:23.364428 26909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:23.372225 26909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:23.372393 26909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:23.372529 26909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:23.372630 26909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:23.372699 26909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:23.383219 26909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:23.384895 26909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:23.386422 26909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:23.399748 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:23.399748 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:23.400175 26917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:23.400486 26909 server_base.cc:1034] running on GCE node
I20260929 16:03:23.401368 26909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:23.402899 26909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:23.404152 26909 hybrid_clock.cc:648] HybridClock initialized: now 1790697803404109 us; error 47 us; skew 500 ppm
I20260929 16:03:23.404527 26909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:23 dist-test-slave-b69v krb5kdc[26896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:23.408118 26909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:23.409703 26909 webserver.cc:460] Webserver started at http://127.0.0.1:35325/ using document root <none> and password file <none>
I20260929 16:03:23.410528 26909 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:23.410696 26909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:23.411054 26909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:23.414525 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "3fbbea99139e478ca8efe327e4691a82"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-b69v"
server_key: "6fe8ec6931d55f7b00169527db6253ff"
server_key_iv: "27720fa18f5908387cc1725cd753a3a0"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.415499 26909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "3fbbea99139e478ca8efe327e4691a82"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-b69v"
server_key: "6fe8ec6931d55f7b00169527db6253ff"
server_key_iv: "27720fa18f5908387cc1725cd753a3a0"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.421530 26909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:03:23.425442 26924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:23.426949 26909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:23.427155 26909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "3fbbea99139e478ca8efe327e4691a82"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-b69v"
server_key: "6fe8ec6931d55f7b00169527db6253ff"
server_key_iv: "27720fa18f5908387cc1725cd753a3a0"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.427393 26909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:23.433936 26909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:23.438711 26909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:23.438997 26909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:23.459086 26909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41013
I20260929 16:03:23.459105 26975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41013 every 8 connection(s)
I20260929 16:03:23.461040 26909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:23.466020 26976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:23.466917 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 26909
I20260929 16:03:23.467137 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:23.467672 5588 external_mini_cluster.cc:1428] Setting key 45c2c6431bff75512a3cbf0df14879d5
I20260929 16:03:23.476773 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.469739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:51523 (local address 127.0.0.1:41013)
0929 16:03:23.470646 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.470660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.470708 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:03:23.472908 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.472934 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.473357 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.473745 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.474992 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.474999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.475013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:23.475027 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.475101 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:23.475123 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.475299 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.475413 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.475644 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":426,"threads_started":1}
I20260929 16:03:23.481109 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:23.481546 26976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82: Bootstrap starting.
I20260929 16:03:23.485631 26976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:23.487262 26976 log.cc:824] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:23.491237 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.487193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:50497 (local address 127.0.0.1:41013)
0929 16:03:23.488429 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.488439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.489052 (+ 613us) server_negotiation.cc:407] Connection header received
0929 16:03:23.489287 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.489293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.489362 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.489495 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.490560 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.490566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.490570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:23.490596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.490638 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:23.490654 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.490777 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.490876 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.490948 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824}
I20260929 16:03:23.491757 26976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82: No bootstrap required, opened a new log
I20260929 16:03:23.500118 26976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } }
I20260929 16:03:23.500543 26976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:23.500624 26976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbbea99139e478ca8efe327e4691a82, State: Initialized, Role: FOLLOWER
I20260929 16:03:23.501302 26976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } }
I20260929 16:03:23.501552 26976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:23.501655 26976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:23.501801 26976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:23.504271 26976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } }
I20260929 16:03:23.504846 26976 leader_election.cc:302] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fbbea99139e478ca8efe327e4691a82; no voters:
I20260929 16:03:23.505466 26976 leader_election.cc:288] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:23.505692 26986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:23.506899 26986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbbea99139e478ca8efe327e4691a82, State: Running, Role: LEADER
I20260929 16:03:23.507531 26986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } }
I20260929 16:03:23.508116 26976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:23.510999 26988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fbbea99139e478ca8efe327e4691a82. Latest consensus state: current_term: 1 leader_uuid: "3fbbea99139e478ca8efe327e4691a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } } }
I20260929 16:03:23.511268 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fbbea99139e478ca8efe327e4691a82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fbbea99139e478ca8efe327e4691a82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41013 } } }
I20260929 16:03:23.511520 26988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:23.511555 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:23.512601 26993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:23.519286 26993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:23.529382 26993 catalog_manager.cc:1347] Generated new cluster ID: e3b72af008a14f6d84e1d40202085010
I20260929 16:03:23.529517 26993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:23.542454 26993 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:23.542928 26993 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:23.553831 26993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fbbea99139e478ca8efe327e4691a82: Generated new TSK 0
I20260929 16:03:23.555155 26993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:23.570221 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26909
W20260929 16:03:23.571982 26971 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.133:51523
W20260929 16:03:23.572853 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 26909: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:23.573019 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 26909
2026-09-29T16:03:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:23 dist-test-slave-b69v krb5kdc[27007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:23 dist-test-slave-b69v krb5kdc[27007](info): set up 2 sockets
Sep 29 16:03:23 dist-test-slave-b69v krb5kdc[27007](info): commencing operation
krb5kdc: starting...
W20260929 16:03:24.713043 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:24 dist-test-slave-b69v krb5kdc[27007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:25.035452 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:25.417987 27020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:25.418282 27020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:25.418380 27020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:25.425486 27020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:25.425585 27020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:25.425628 27020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:25.425709 27020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:25.425752 27020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:25.436580 27020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:25.438376 27020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:25.440069 27020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:25.454991 27025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:25.455047 27026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:25.454948 27028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:25.457760 27020 server_base.cc:1034] running on GCE node
I20260929 16:03:25.458731 27020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:25.460608 27020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:25.461833 27020 hybrid_clock.cc:648] HybridClock initialized: now 1790697805461792 us; error 37 us; skew 500 ppm
I20260929 16:03:25.462220 27020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:25 dist-test-slave-b69v krb5kdc[27007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:25.465915 27020 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:25.468019 27020 webserver.cc:460] Webserver started at http://127.0.0.1:41683/ using document root <none> and password file <none>
I20260929 16:03:25.468940 27020 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:25.469053 27020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:25.469333 27020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:25.473016 27020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "2af2e7f4bc844a0fba2dd860a046cd5a"
format_stamp: "Formatted at 2026-09-29 16:03:25 on dist-test-slave-b69v"
server_key: "02fc300e34d8fdecfba0f4e98dc96fbe"
server_key_iv: "5a78df3a8607956c1b4aecb814179b30"
server_key_version: "encryptionkey@0"
I20260929 16:03:25.473850 27020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "2af2e7f4bc844a0fba2dd860a046cd5a"
format_stamp: "Formatted at 2026-09-29 16:03:25 on dist-test-slave-b69v"
server_key: "02fc300e34d8fdecfba0f4e98dc96fbe"
server_key_iv: "5a78df3a8607956c1b4aecb814179b30"
server_key_version: "encryptionkey@0"
I20260929 16:03:25.480239 27020 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:03:25.483877 27035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:25.485646 27020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:25.485864 27020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "2af2e7f4bc844a0fba2dd860a046cd5a"
format_stamp: "Formatted at 2026-09-29 16:03:25 on dist-test-slave-b69v"
server_key: "02fc300e34d8fdecfba0f4e98dc96fbe"
server_key_iv: "5a78df3a8607956c1b4aecb814179b30"
server_key_version: "encryptionkey@0"
I20260929 16:03:25.486053 27020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:25.521010 27020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:25.526217 27020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:25.526585 27020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:25.547816 27020 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45485
I20260929 16:03:25.547866 27086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45485 every 8 connection(s)
I20260929 16:03:25.549463 27020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:25.553730 27087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:25.559897 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27020
I20260929 16:03:25.560159 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:25.560645 5588 external_mini_cluster.cc:1428] Setting key 28d61a241ef2d7c6d18adec3a7e34594
I20260929 16:03:25.565359 27087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a: Bootstrap starting.
I20260929 16:03:25.569254 27087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:25.570648 27087 log.cc:824] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:25.574347 27087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a: No bootstrap required, opened a new log
Sep 29 16:03:25 dist-test-slave-b69v krb5kdc[27007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:03:25.582743 27087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260929 16:03:25.583169 27087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:25.583300 27087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2e7f4bc844a0fba2dd860a046cd5a, State: Initialized, Role: FOLLOWER
I20260929 16:03:25.584112 27087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260929 16:03:25.584400 27087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:25.584566 27087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:25.584758 27087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:25.585146 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.562725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:60085 (local address 127.0.0.1:45485)
0929 16:03:25.563616 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.563627 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.563673 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:03:25.565399 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.565421 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.565853 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.566161 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:25.568125 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.569394 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.571313 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.572412 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.572562 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.577523 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:25.577531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:25.577545 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:25.577578 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:25.580749 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:25.581544 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:25.581550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:25.581554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:25.581712 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:25.582207 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:25.582216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:25.582218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:25.582731 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:25.583102 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.583580 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.583896 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":458,"threads_started":1}
I20260929 16:03:25.587193 27087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260929 16:03:25.587806 27087 leader_election.cc:302] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2af2e7f4bc844a0fba2dd860a046cd5a; no voters:
I20260929 16:03:25.588579 27087 leader_election.cc:288] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:25.588681 27092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:25.589255 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:25.591406 27087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:25.592490 27092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [term 1 LEADER]: Becoming Leader. State: Replica: 2af2e7f4bc844a0fba2dd860a046cd5a, State: Running, Role: LEADER
I20260929 16:03:25.593313 27092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260929 16:03:25.597635 27094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2af2e7f4bc844a0fba2dd860a046cd5a. Latest consensus state: current_term: 1 leader_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } } }
I20260929 16:03:25.598160 27094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:25.598843 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2e7f4bc844a0fba2dd860a046cd5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } } }
I20260929 16:03:25.599205 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:25.614265 27100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:25.620996 27100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:25.635218 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.610198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:49459 (local address 127.0.0.1:45485)
0929 16:03:25.610475 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.610486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.612759 (+ 2273us) server_negotiation.cc:407] Connection header received
0929 16:03:25.613432 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.613443 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.613565 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.613673 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:25.616309 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.617142 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.619057 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.619809 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.619945 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.624305 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:25.624319 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:25.624327 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:25.624382 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:25.629594 (+ 5212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:25.630857 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:25.630866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:25.630873 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:25.631064 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:25.631771 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:25.631779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:25.631784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:25.632481 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:25.632849 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.634387 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.634666 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:03:25.639122 27100 catalog_manager.cc:1347] Generated new cluster ID: 7ce083bf41b04e31a90eafb83811914b
I20260929 16:03:25.639312 27100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:25.652305 27100 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:25.653721 27100 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:25.664772 27100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2af2e7f4bc844a0fba2dd860a046cd5a: Generated new TSK 0
I20260929 16:03:25.665911 27100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:25.675915 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27020
I20260929 16:03:25.677032 27082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:25.771418 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27020
W20260929 16:03:25.791028 27005 connection.cc:535] client connection to 127.0.0.1:45485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45485): Connection reset by peer (error 104)
2026-09-29T16:03:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:25 dist-test-slave-b69v krb5kdc[27120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:25 dist-test-slave-b69v krb5kdc[27120](info): set up 2 sockets
Sep 29 16:03:25 dist-test-slave-b69v krb5kdc[27120](info): commencing operation
krb5kdc: starting...
W20260929 16:03:26.881016 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:27 dist-test-slave-b69v krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:27.181639 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:27.543373 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:27.543955 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:27.544234 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:27.552997 27133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:27.553251 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:27.553371 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:27.553503 27133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:27.553593 27133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:27.566005 27133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:27.568029 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:27.569936 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:27.586055 27138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:27.586133 27141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:27.585986 27139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:27.587568 27133 server_base.cc:1034] running on GCE node
I20260929 16:03:27.588792 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:27.591023 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:27.592407 27133 hybrid_clock.cc:648] HybridClock initialized: now 1790697807592253 us; error 114 us; skew 500 ppm
I20260929 16:03:27.592999 27133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:27 dist-test-slave-b69v krb5kdc[27120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:27.605819 27133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:27.607766 27133 webserver.cc:460] Webserver started at http://127.0.0.1:35011/ using document root <none> and password file <none>
I20260929 16:03:27.608671 27133 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:27.608788 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:27.609093 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:27.612775 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "322d16b6badd4c02b07abd68f2f9d02f"
format_stamp: "Formatted at 2026-09-29 16:03:27 on dist-test-slave-b69v"
server_key: "9470bc465b26cf763d5e382278b4791f"
server_key_iv: "b2c20cd978582497b1d3a1710b021ebb"
server_key_version: "encryptionkey@0"
I20260929 16:03:27.613607 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "322d16b6badd4c02b07abd68f2f9d02f"
format_stamp: "Formatted at 2026-09-29 16:03:27 on dist-test-slave-b69v"
server_key: "9470bc465b26cf763d5e382278b4791f"
server_key_iv: "b2c20cd978582497b1d3a1710b021ebb"
server_key_version: "encryptionkey@0"
I20260929 16:03:27.619445 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:03:27.624176 27148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:27.625722 27133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:27.625875 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "322d16b6badd4c02b07abd68f2f9d02f"
format_stamp: "Formatted at 2026-09-29 16:03:27 on dist-test-slave-b69v"
server_key: "9470bc465b26cf763d5e382278b4791f"
server_key_iv: "b2c20cd978582497b1d3a1710b021ebb"
server_key_version: "encryptionkey@0"
I20260929 16:03:27.626046 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:27.650122 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:27.654664 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:27.654919 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:27.675523 27133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38359
I20260929 16:03:27.675557 27199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38359 every 8 connection(s)
I20260929 16:03:27.676901 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:27.680886 27200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:27.685415 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27133
I20260929 16:03:27.685622 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:27.686136 5588 external_mini_cluster.cc:1428] Setting key be5a966c710ce55c17741208529e5335
I20260929 16:03:27.692662 27200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f: Bootstrap starting.
I20260929 16:03:27.697167 27200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:27.698437 27200 log.cc:824] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:27.701526 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:27.687998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:55445 (local address 127.0.0.1:38359)
0929 16:03:27.688801 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:27.688812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:27.688853 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:03:27.690759 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:27.690783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:27.691203 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:27.691700 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:27.692380 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:27.694481 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:27.696180 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:27.697211 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:27.697390 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:27.698621 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:27.698631 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:27.698650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:27.699419 (+ 769us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:27.699501 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:27.699532 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:27.699649 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:27.699963 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:03:27.700171 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":335,"threads_started":1}
I20260929 16:03:27.701990 27200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f: No bootstrap required, opened a new log
I20260929 16:03:27.705773 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:27.710179 27200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:03:27.710801 27200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:27.710929 27200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322d16b6badd4c02b07abd68f2f9d02f, State: Initialized, Role: FOLLOWER
I20260929 16:03:27.711669 27200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:03:27.711989 27200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:27.712162 27200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:27.712427 27200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:27.715647 27200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:03:27.716282 27200 leader_election.cc:302] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 322d16b6badd4c02b07abd68f2f9d02f; no voters:
I20260929 16:03:27.717073 27200 leader_election.cc:288] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:27.717177 27210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:27.717584 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:27.710576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:60927 (local address 127.0.0.1:38359)
0929 16:03:27.710791 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:27.710797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:27.711135 (+ 338us) server_negotiation.cc:407] Connection header received
0929 16:03:27.711364 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:27.711372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:27.711455 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:27.711539 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:27.712216 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:27.713127 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:27.714955 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:27.715411 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:27.715517 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:27.716695 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:27.716699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:27.716703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:27.716719 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:27.716781 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:27.716798 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:27.716927 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:27.717107 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 16:03:27.717166 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:03:27.718904 27210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [term 1 LEADER]: Becoming Leader. State: Replica: 322d16b6badd4c02b07abd68f2f9d02f, State: Running, Role: LEADER
I20260929 16:03:27.719543 27210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } }
I20260929 16:03:27.719705 27200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:27.722965 27212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 322d16b6badd4c02b07abd68f2f9d02f. Latest consensus state: current_term: 1 leader_uuid: "322d16b6badd4c02b07abd68f2f9d02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 16:03:27.723510 27212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:27.723600 27211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "322d16b6badd4c02b07abd68f2f9d02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "322d16b6badd4c02b07abd68f2f9d02f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38359 } } }
I20260929 16:03:27.723912 27211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:27.727732 27218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:27.733631 27218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:27.744812 27218 catalog_manager.cc:1347] Generated new cluster ID: 6ca6656121d942edb6bceffe4101eee9
I20260929 16:03:27.745005 27218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:27.760514 27218 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:27.762216 27218 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:27.769330 27218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 322d16b6badd4c02b07abd68f2f9d02f: Generated new TSK 0
I20260929 16:03:27.770448 27218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:27.781726 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27133
W20260929 16:03:27.783207 27195 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.133:55445
W20260929 16:03:27.784140 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:27.784312 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27133
W20260929 16:03:27.807533 27118 connection.cc:535] client connection to 127.0.0.1:38359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38359): Connection reset by peer (error 104)
2026-09-29T16:03:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:27 dist-test-slave-b69v krb5kdc[27231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:27 dist-test-slave-b69v krb5kdc[27231](info): set up 2 sockets
Sep 29 16:03:27 dist-test-slave-b69v krb5kdc[27231](info): commencing operation
krb5kdc: starting...
W20260929 16:03:28.892853 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:29 dist-test-slave-b69v krb5kdc[27231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:29.191610 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:29.565740 27244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:29.566045 27244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:29.566207 27244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:29.573462 27244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:29.573566 27244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:29.573680 27244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:29.573766 27244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:29.573848 27244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:29.583436 27244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:29.584807 27244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:29.586211 27244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:29.599110 27249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:29.599514 27250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:29.599540 27252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:29.600520 27244 server_base.cc:1034] running on GCE node
I20260929 16:03:29.601301 27244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:29.602721 27244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:29.603957 27244 hybrid_clock.cc:648] HybridClock initialized: now 1790697809603912 us; error 45 us; skew 500 ppm
I20260929 16:03:29.604403 27244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:29 dist-test-slave-b69v krb5kdc[27231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:29.607700 27244 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:29.609378 27244 webserver.cc:460] Webserver started at http://127.0.0.1:42819/ using document root <none> and password file <none>
I20260929 16:03:29.610133 27244 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:29.610294 27244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:29.610635 27244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:29.613916 27244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "dac62fb51ea34878a3a39796c6708ba9"
format_stamp: "Formatted at 2026-09-29 16:03:29 on dist-test-slave-b69v"
server_key: "c36e39a7df727d689f1a8ff7b890365a"
server_key_iv: "ac94e6e439a0cde298fb78d150a5db2a"
server_key_version: "encryptionkey@0"
I20260929 16:03:29.614742 27244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "dac62fb51ea34878a3a39796c6708ba9"
format_stamp: "Formatted at 2026-09-29 16:03:29 on dist-test-slave-b69v"
server_key: "c36e39a7df727d689f1a8ff7b890365a"
server_key_iv: "ac94e6e439a0cde298fb78d150a5db2a"
server_key_version: "encryptionkey@0"
I20260929 16:03:29.620694 27244 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:03:29.624663 27259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:29.626058 27244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:03:29.626266 27244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "dac62fb51ea34878a3a39796c6708ba9"
format_stamp: "Formatted at 2026-09-29 16:03:29 on dist-test-slave-b69v"
server_key: "c36e39a7df727d689f1a8ff7b890365a"
server_key_iv: "ac94e6e439a0cde298fb78d150a5db2a"
server_key_version: "encryptionkey@0"
I20260929 16:03:29.626492 27244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:29.632952 27244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:29.637636 27244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:29.637970 27244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:29.659307 27244 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37263
I20260929 16:03:29.659348 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37263 every 8 connection(s)
I20260929 16:03:29.661023 27244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:29.661506 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27244
I20260929 16:03:29.661720 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:29.662415 5588 external_mini_cluster.cc:1428] Setting key e944138df5585742b530a5dd92ba1c70
I20260929 16:03:29.666393 27311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:29.672446 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:29.664494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:53699 (local address 127.0.0.1:37263)
0929 16:03:29.665398 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:29.665414 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:29.665471 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:03:29.667987 (+ 2516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:29.668015 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:29.668471 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:29.669204 (+ 733us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:29.670424 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:29.670438 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:29.670453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:29.670470 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:29.670568 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:29.670601 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:29.670836 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:29.670981 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:03:29.671174 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":336,"threads_started":1}
I20260929 16:03:29.676419 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:29.680281 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9: Bootstrap starting.
I20260929 16:03:29.683947 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:29.681256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:60915 (local address 127.0.0.1:37263)
0929 16:03:29.681468 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:29.681477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:29.682040 (+ 563us) server_negotiation.cc:407] Connection header received
0929 16:03:29.682186 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:29.682189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:29.682230 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:29.682317 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:29.683261 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:29.683265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:29.683267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:29.683281 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:29.683324 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:29.683339 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:29.683425 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:29.683580 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:03:29.683624 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:03:29.684415 27311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:29.685953 27311 log.cc:824] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:29.688668 27311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9: No bootstrap required, opened a new log
I20260929 16:03:29.696852 27311 raft_consensus.cc:348] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } }
I20260929 16:03:29.697193 27311 raft_consensus.cc:374] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:29.697268 27311 raft_consensus.cc:729] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac62fb51ea34878a3a39796c6708ba9, State: Initialized, Role: FOLLOWER
I20260929 16:03:29.697746 27311 consensus_queue.cc:260] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } }
I20260929 16:03:29.697942 27311 raft_consensus.cc:388] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:29.698060 27311 raft_consensus.cc:482] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:29.698189 27311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:29.700708 27311 raft_consensus.cc:504] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } }
I20260929 16:03:29.701166 27311 leader_election.cc:302] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dac62fb51ea34878a3a39796c6708ba9; no voters:
I20260929 16:03:29.701983 27311 leader_election.cc:288] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:29.701996 27321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:29.702847 27321 raft_consensus.cc:686] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [term 1 LEADER]: Becoming Leader. State: Replica: dac62fb51ea34878a3a39796c6708ba9, State: Running, Role: LEADER
I20260929 16:03:29.703552 27321 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } }
I20260929 16:03:29.704916 27311 sys_catalog.cc:564] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:29.706598 27322 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac62fb51ea34878a3a39796c6708ba9. Latest consensus state: current_term: 1 leader_uuid: "dac62fb51ea34878a3a39796c6708ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } } }
I20260929 16:03:29.707180 27322 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:29.706640 27323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dac62fb51ea34878a3a39796c6708ba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac62fb51ea34878a3a39796c6708ba9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37263 } } }
I20260929 16:03:29.707760 27323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:29.710716 27327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:29.716468 27327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:29.724967 27327 catalog_manager.cc:1347] Generated new cluster ID: 7719c153f4654aa0a1168c859153a815
I20260929 16:03:29.725076 27327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:29.748216 27327 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:29.748705 27327 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:29.757230 27327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dac62fb51ea34878a3a39796c6708ba9: Generated new TSK 0
I20260929 16:03:29.757964 27327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:29.761909 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27244
W20260929 16:03:29.763084 27306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.133:53699
W20260929 16:03:29.763761 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:29.763896 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27244
2026-09-29T16:03:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:29 dist-test-slave-b69v krb5kdc[27342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:29 dist-test-slave-b69v krb5kdc[27342](info): set up 2 sockets
Sep 29 16:03:29 dist-test-slave-b69v krb5kdc[27342](info): commencing operation
krb5kdc: starting...
W20260929 16:03:30.905071 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:31 dist-test-slave-b69v krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:31.239567 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34325
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:31.628958 27355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:31.629263 27355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:31.629472 27355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:31.636734 27355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:31.636874 27355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:31.636972 27355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:31.637087 27355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:31.637169 27355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:31.646487 27355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:34325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:31.647939 27355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:31.649464 27355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:31.662627 27360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:31.662578 27363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:31.662604 27361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:31.663713 27355 server_base.cc:1034] running on GCE node
I20260929 16:03:31.664883 27355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:31.666497 27355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:31.667728 27355 hybrid_clock.cc:648] HybridClock initialized: now 1790697811667694 us; error 49 us; skew 500 ppm
I20260929 16:03:31.668140 27355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:31 dist-test-slave-b69v krb5kdc[27342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697811, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:31.671576 27355 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:31.673189 27355 webserver.cc:460] Webserver started at http://127.5.117.62:44915/ using document root <none> and password file <none>
I20260929 16:03:31.673970 27355 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:31.674140 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:31.674456 27355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:31.677728 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "750951c0f8df4bea8af06c8c3510b9df"
format_stamp: "Formatted at 2026-09-29 16:03:31 on dist-test-slave-b69v"
server_key: "75e533ca947007bbda0314d12f721060"
server_key_iv: "8d34897a6273223a39baf0402171bf8a"
server_key_version: "encryptionkey@0"
I20260929 16:03:31.678555 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "750951c0f8df4bea8af06c8c3510b9df"
format_stamp: "Formatted at 2026-09-29 16:03:31 on dist-test-slave-b69v"
server_key: "75e533ca947007bbda0314d12f721060"
server_key_iv: "8d34897a6273223a39baf0402171bf8a"
server_key_version: "encryptionkey@0"
I20260929 16:03:31.684335 27355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:03:31.688405 27370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:31.690286 27355 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:03:31.690593 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "750951c0f8df4bea8af06c8c3510b9df"
format_stamp: "Formatted at 2026-09-29 16:03:31 on dist-test-slave-b69v"
server_key: "75e533ca947007bbda0314d12f721060"
server_key_iv: "8d34897a6273223a39baf0402171bf8a"
server_key_version: "encryptionkey@0"
I20260929 16:03:31.690969 27355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:31.706948 27355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:31.711660 27355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:31.711973 27355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:31.733944 27355 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:34325
I20260929 16:03:31.733986 27421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:34325 every 8 connection(s)
I20260929 16:03:31.736176 27355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:31.740832 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:31.744861 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27355
I20260929 16:03:31.745081 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:31.745610 5588 external_mini_cluster.cc:1428] Setting key 5fcf19e0be5a2d91f0293efb05583a4a
I20260929 16:03:31.754276 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df: Bootstrap starting.
I20260929 16:03:31.758399 27422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:31.759802 27422 log.cc:824] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:03:31 dist-test-slave-b69v krb5kdc[27342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:31.762842 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df: No bootstrap required, opened a new log
I20260929 16:03:31.769858 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:31.747475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46775 (local address 127.5.117.62:34325)
0929 16:03:31.748325 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:31.748337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:31.748381 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:03:31.750064 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:31.750086 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:31.750481 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:31.750832 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:31.752733 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:31.754061 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:31.755815 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:31.756765 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:31.756898 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:31.762322 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:31.762329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:31.762342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:31.762376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:31.765655 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:31.766498 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:31.766504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:31.766510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:31.766699 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:31.767190 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:31.767194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:31.767196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:31.767709 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:31.768051 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:31.768568 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 16:03:31.768850 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":431,"threads_started":1}
I20260929 16:03:31.770706 27422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } }
I20260929 16:03:31.771142 27422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:31.771278 27422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750951c0f8df4bea8af06c8c3510b9df, State: Initialized, Role: FOLLOWER
I20260929 16:03:31.771989 27422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } }
I20260929 16:03:31.772342 27422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:31.772539 27422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:31.772722 27422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:31.773981 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:31.775157 27422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } }
I20260929 16:03:31.775609 27422 leader_election.cc:302] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 750951c0f8df4bea8af06c8c3510b9df; no voters:
I20260929 16:03:31.776350 27422 leader_election.cc:288] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:31.776477 27427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:31.780246 27422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:31.780359 27427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [term 1 LEADER]: Becoming Leader. State: Replica: 750951c0f8df4bea8af06c8c3510b9df, State: Running, Role: LEADER
I20260929 16:03:31.781031 27427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } }
I20260929 16:03:31.792958 27430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "750951c0f8df4bea8af06c8c3510b9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } } }
I20260929 16:03:31.793445 27430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:31.795790 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:31.779165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34647 (local address 127.5.117.62:34325)
0929 16:03:31.779289 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:31.779294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:31.779831 (+ 537us) server_negotiation.cc:407] Connection header received
0929 16:03:31.780017 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:31.780023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:31.780124 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:31.780207 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:31.782167 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:31.783438 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:31.785029 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:31.785483 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:31.785611 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:31.788169 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:31.788174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:31.788179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:31.788213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:31.791144 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:31.792165 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:31.792171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:31.792173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:31.792397 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:31.793183 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:31.793191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:31.793200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:31.793756 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:31.794143 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:31.795029 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 16:03:31.795270 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 16:03:31.799489 27431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 750951c0f8df4bea8af06c8c3510b9df. Latest consensus state: current_term: 1 leader_uuid: "750951c0f8df4bea8af06c8c3510b9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750951c0f8df4bea8af06c8c3510b9df" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 34325 } } }
I20260929 16:03:31.799820 27431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:31.799850 27448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:31.804492 27448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:31.812606 27448 catalog_manager.cc:1347] Generated new cluster ID: 1f03b97c60b4474e88dbef549671508d
I20260929 16:03:31.812713 27448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:31.826704 27448 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:31.827803 27448 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:31.838147 27448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 750951c0f8df4bea8af06c8c3510b9df: Generated new TSK 0
I20260929 16:03:31.838842 27448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:31.844799 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27355
I20260929 16:03:31.846304 27417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:31.938633 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27355
W20260929 16:03:31.961772 27340 connection.cc:535] client connection to 127.5.117.62:34325 recv error: Network error: failed to read from TLS socket (remote: 127.5.117.62:34325): Connection reset by peer (error 104)
2026-09-29T16:03:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:32 dist-test-slave-b69v krb5kdc[27455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:32 dist-test-slave-b69v krb5kdc[27455](info): set up 2 sockets
Sep 29 16:03:32 dist-test-slave-b69v krb5kdc[27455](info): commencing operation
krb5kdc: starting...
W20260929 16:03:33.068766 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:33 dist-test-slave-b69v krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:33.387352 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45479
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:45479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:33.771894 27468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:33.772361 27468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:33.772549 27468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:33.779901 27468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:33.780082 27468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:33.780184 27468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:33.780311 27468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:33.780398 27468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:33.790560 27468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:45479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:33.792181 27468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:33.793730 27468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:33.807200 27473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:33.807243 27474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:33.807277 27476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:33.808440 27468 server_base.cc:1034] running on GCE node
I20260929 16:03:33.809199 27468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:33.810684 27468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:33.812168 27468 hybrid_clock.cc:648] HybridClock initialized: now 1790697813811992 us; error 29 us; skew 500 ppm
I20260929 16:03:33.812878 27468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:33 dist-test-slave-b69v krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697813, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:33.824210 27468 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:33.825881 27468 webserver.cc:460] Webserver started at http://127.5.117.62:34935/ using document root <none> and password file <none>
I20260929 16:03:33.826643 27468 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:33.826809 27468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:33.827144 27468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:33.830380 27468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "9dcb303f012944b6ab9eace5d1d3f712"
format_stamp: "Formatted at 2026-09-29 16:03:33 on dist-test-slave-b69v"
server_key: "92014cd10ac11de2aff80211154baa60"
server_key_iv: "e10baf32f08f951eb2adff08e2b99be3"
server_key_version: "encryptionkey@0"
I20260929 16:03:33.831214 27468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "9dcb303f012944b6ab9eace5d1d3f712"
format_stamp: "Formatted at 2026-09-29 16:03:33 on dist-test-slave-b69v"
server_key: "92014cd10ac11de2aff80211154baa60"
server_key_iv: "e10baf32f08f951eb2adff08e2b99be3"
server_key_version: "encryptionkey@0"
I20260929 16:03:33.836658 27468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:03:33.840276 27483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:33.842083 27468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:33.842275 27468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "9dcb303f012944b6ab9eace5d1d3f712"
format_stamp: "Formatted at 2026-09-29 16:03:33 on dist-test-slave-b69v"
server_key: "92014cd10ac11de2aff80211154baa60"
server_key_iv: "e10baf32f08f951eb2adff08e2b99be3"
server_key_version: "encryptionkey@0"
I20260929 16:03:33.842496 27468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:33.867334 27468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:33.872304 27468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:33.872633 27468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:33.894378 27468 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:45479
I20260929 16:03:33.894421 27534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:45479 every 8 connection(s)
I20260929 16:03:33.895888 27468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:33.901103 27535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:33.902601 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27468
I20260929 16:03:33.902812 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:33.903477 5588 external_mini_cluster.cc:1428] Setting key b82b66fb20eb37c885d2283b3f61804a
I20260929 16:03:33.916550 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: Bootstrap starting.
I20260929 16:03:33.920892 27535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:33.922355 27535 log.cc:824] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:33.922417 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:33.906342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55457 (local address 127.5.117.62:45479)
0929 16:03:33.907676 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:33.907694 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:33.908285 (+ 591us) server_negotiation.cc:407] Connection header received
0929 16:03:33.911004 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:33.911039 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:33.911652 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:33.912243 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:33.912854 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:33.915041 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:33.916723 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:33.917750 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:33.917945 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:33.919340 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:33.919350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:33.919368 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:33.920190 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:33.920284 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:33.920316 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:33.920489 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:33.920769 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 16:03:33.920984 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":621,"threads_started":1}
I20260929 16:03:33.925885 27535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: No bootstrap required, opened a new log
I20260929 16:03:33.927078 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:33.934630 27535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } }
I20260929 16:03:33.935084 27535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:33.935242 27535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcb303f012944b6ab9eace5d1d3f712, State: Initialized, Role: FOLLOWER
I20260929 16:03:33.935896 27535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } }
I20260929 16:03:33.936224 27535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:33.936389 27535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:33.936581 27535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:33.939739 27535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } }
I20260929 16:03:33.940434 27535 leader_election.cc:302] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dcb303f012944b6ab9eace5d1d3f712; no voters:
I20260929 16:03:33.940876 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:33.932996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32931 (local address 127.5.117.62:45479)
0929 16:03:33.933212 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:33.933218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:33.933887 (+ 669us) server_negotiation.cc:407] Connection header received
0929 16:03:33.933966 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:33.933970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:33.934025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:33.934105 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:33.934782 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:33.935906 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:33.937928 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:33.938408 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:33.938515 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:33.939874 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:33.939916 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:33.939927 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:33.939956 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:33.940032 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:33.940105 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:33.940217 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:33.940446 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:03:33.940491 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:03:33.941251 27535 leader_election.cc:288] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:33.941396 27545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:33.943459 27535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:33.947000 27545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [term 1 LEADER]: Becoming Leader. State: Replica: 9dcb303f012944b6ab9eace5d1d3f712, State: Running, Role: LEADER
I20260929 16:03:33.947646 27545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } }
I20260929 16:03:33.951081 27546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dcb303f012944b6ab9eace5d1d3f712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } } }
I20260929 16:03:33.951579 27546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:33.952517 27547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcb303f012944b6ab9eace5d1d3f712. Latest consensus state: current_term: 1 leader_uuid: "9dcb303f012944b6ab9eace5d1d3f712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcb303f012944b6ab9eace5d1d3f712" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 45479 } } }
I20260929 16:03:33.953011 27547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712 [sys.catalog]: This master's current role is: LEADER
W20260929 16:03:33.956681 27559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:33.956784 27559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:33.956987 27560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:33.959301 27560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:33.968174 27560 catalog_manager.cc:1347] Generated new cluster ID: e4ebfb61c81d4c23ab5387ff6ba4eb7c
I20260929 16:03:33.968290 27560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:33.979490 27560 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:33.981405 27560 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:34.004702 27560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dcb303f012944b6ab9eace5d1d3f712: Generated new TSK 0
I20260929 16:03:34.005857 27560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:34.017912 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27468
W20260929 16:03:34.019392 27530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55457
W20260929 16:03:34.020195 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:34.020327 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27468
W20260929 16:03:34.041797 27453 connection.cc:535] client connection to 127.5.117.62:45479 recv error: Network error: failed to read from TLS socket (remote: 127.5.117.62:45479): Connection reset by peer (error 104)
2026-09-29T16:03:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2081 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:34 dist-test-slave-b69v krb5kdc[27566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:34 dist-test-slave-b69v krb5kdc[27566](info): set up 2 sockets
Sep 29 16:03:34 dist-test-slave-b69v krb5kdc[27566](info): commencing operation
krb5kdc: starting...
W20260929 16:03:35.144972 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:35 dist-test-slave-b69v krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:35.445858 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:33789
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:33789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:35.826037 27579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:35.826449 27579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:35.826637 27579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:35.834299 27579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:35.834472 27579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:35.834591 27579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:35.834710 27579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:35.834793 27579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:35.845532 27579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:33789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:33789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:35.847190 27579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:35.848785 27579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:35.862401 27584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:35.862574 27587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:35.862537 27585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:35.863608 27579 server_base.cc:1034] running on GCE node
I20260929 16:03:35.864837 27579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:35.866394 27579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:35.867650 27579 hybrid_clock.cc:648] HybridClock initialized: now 1790697815867625 us; error 57 us; skew 500 ppm
I20260929 16:03:35.868202 27579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:35 dist-test-slave-b69v krb5kdc[27566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697815, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:35.871552 27579 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:35.873731 27579 webserver.cc:460] Webserver started at http://127.5.117.62:35931/ using document root <none> and password file <none>
I20260929 16:03:35.874447 27579 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:35.874557 27579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:35.874900 27579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:35.878521 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "ce0ae3d974f74582bf503937d5726952"
format_stamp: "Formatted at 2026-09-29 16:03:35 on dist-test-slave-b69v"
server_key: "4adba159f3eade22874b0d151ab1972e"
server_key_iv: "83a4f94ac1e374aa11a5541e7c009780"
server_key_version: "encryptionkey@0"
I20260929 16:03:35.879446 27579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "ce0ae3d974f74582bf503937d5726952"
format_stamp: "Formatted at 2026-09-29 16:03:35 on dist-test-slave-b69v"
server_key: "4adba159f3eade22874b0d151ab1972e"
server_key_iv: "83a4f94ac1e374aa11a5541e7c009780"
server_key_version: "encryptionkey@0"
I20260929 16:03:35.884953 27579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:03:35.888547 27594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:35.889961 27579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:03:35.890174 27579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "ce0ae3d974f74582bf503937d5726952"
format_stamp: "Formatted at 2026-09-29 16:03:35 on dist-test-slave-b69v"
server_key: "4adba159f3eade22874b0d151ab1972e"
server_key_iv: "83a4f94ac1e374aa11a5541e7c009780"
server_key_version: "encryptionkey@0"
I20260929 16:03:35.890401 27579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:35.896658 27579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:35.901266 27579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:35.901587 27579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:35.924340 27579 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:33789
I20260929 16:03:35.924425 27645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:33789 every 8 connection(s)
I20260929 16:03:35.925853 27579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:35.929930 27646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:35.934799 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27579
I20260929 16:03:35.935010 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:35.935550 5588 external_mini_cluster.cc:1428] Setting key 60f18b73d9c0f408ad61273f309bbd04
I20260929 16:03:35.942035 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952: Bootstrap starting.
I20260929 16:03:35.944695 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:35.937465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54673 (local address 127.5.117.62:33789)
0929 16:03:35.938340 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:35.938352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:35.938400 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:03:35.940474 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:35.940505 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:35.940822 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:35.941146 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:35.942448 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:35.942459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:35.942479 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:35.942509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:35.942646 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:35.942680 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:35.942968 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:35.943174 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 16:03:35.943448 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":443,"threads_started":1}
I20260929 16:03:35.945675 27646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:35.947140 27646 log.cc:824] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:35.948824 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:35.950592 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952: No bootstrap required, opened a new log
I20260929 16:03:35.957516 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:35.954555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48803 (local address 127.5.117.62:33789)
0929 16:03:35.954783 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:35.954791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:35.955127 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:03:35.955304 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:35.955307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:35.955356 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:35.955453 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:35.956790 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:35.956794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:35.956798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:35.956813 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:35.956867 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:35.956886 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:35.956994 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:35.957132 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:03:35.957181 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:03:35.960543 27646 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } }
I20260929 16:03:35.960902 27646 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:35.961000 27646 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0ae3d974f74582bf503937d5726952, State: Initialized, Role: FOLLOWER
I20260929 16:03:35.961612 27646 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } }
I20260929 16:03:35.961862 27646 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:35.962018 27646 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:35.962211 27646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:35.964692 27646 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } }
I20260929 16:03:35.965128 27646 leader_election.cc:302] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0ae3d974f74582bf503937d5726952; no voters:
I20260929 16:03:35.965648 27646 leader_election.cc:288] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:35.965790 27656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:35.966530 27656 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [term 1 LEADER]: Becoming Leader. State: Replica: ce0ae3d974f74582bf503937d5726952, State: Running, Role: LEADER
I20260929 16:03:35.967057 27656 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } }
I20260929 16:03:35.967808 27646 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:35.970126 27658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0ae3d974f74582bf503937d5726952. Latest consensus state: current_term: 1 leader_uuid: "ce0ae3d974f74582bf503937d5726952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } } }
I20260929 16:03:35.970597 27657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0ae3d974f74582bf503937d5726952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ae3d974f74582bf503937d5726952" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 33789 } } }
I20260929 16:03:35.970816 27658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:35.970896 27657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:35.976459 27665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:35.982012 27665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:35.991719 27665 catalog_manager.cc:1347] Generated new cluster ID: 4c0edb7b6aba4a91bc73b5d0282968f7
I20260929 16:03:35.991843 27665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:36.019806 27665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:36.020316 27665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:36.028563 27665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0ae3d974f74582bf503937d5726952: Generated new TSK 0
I20260929 16:03:36.029373 27665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:36.039309 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27579
W20260929 16:03:36.040400 27641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54673
W20260929 16:03:36.041091 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27579: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:36.041283 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27579
2026-09-29T16:03:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:36 dist-test-slave-b69v krb5kdc[27677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:36 dist-test-slave-b69v krb5kdc[27677](info): set up 2 sockets
Sep 29 16:03:36 dist-test-slave-b69v krb5kdc[27677](info): commencing operation
krb5kdc: starting...
W20260929 16:03:37.144897 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:37 dist-test-slave-b69v krb5kdc[27677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:37.450119 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35471
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:35471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:37.830106 27690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:37.830454 27690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:37.830660 27690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:37.838739 27690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:37.838847 27690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:37.838896 27690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:37.838987 27690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:37.839066 27690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:37.849399 27690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:35471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:35471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:37.850904 27690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:37.852444 27690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:37.865573 27696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:37.865651 27698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:37.865617 27695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:37.866693 27690 server_base.cc:1034] running on GCE node
I20260929 16:03:37.867522 27690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:37.869192 27690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:37.870442 27690 hybrid_clock.cc:648] HybridClock initialized: now 1790697817870388 us; error 65 us; skew 500 ppm
I20260929 16:03:37.870823 27690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:37 dist-test-slave-b69v krb5kdc[27677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697817, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:37.874635 27690 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:37.876281 27690 webserver.cc:460] Webserver started at http://127.5.117.62:38633/ using document root <none> and password file <none>
I20260929 16:03:37.877084 27690 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:37.877244 27690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:37.877579 27690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:37.880918 27690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "0ccf08cacca0477babf708de41acf39d"
format_stamp: "Formatted at 2026-09-29 16:03:37 on dist-test-slave-b69v"
server_key: "1e6f38e1d0a570b817b238c674bc8804"
server_key_iv: "7b69478d9e74e5ebb3b0efda2e5f0676"
server_key_version: "encryptionkey@0"
I20260929 16:03:37.881798 27690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "0ccf08cacca0477babf708de41acf39d"
format_stamp: "Formatted at 2026-09-29 16:03:37 on dist-test-slave-b69v"
server_key: "1e6f38e1d0a570b817b238c674bc8804"
server_key_iv: "7b69478d9e74e5ebb3b0efda2e5f0676"
server_key_version: "encryptionkey@0"
I20260929 16:03:37.887602 27690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:03:37.891708 27705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:37.893155 27690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:37.893359 27690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "0ccf08cacca0477babf708de41acf39d"
format_stamp: "Formatted at 2026-09-29 16:03:37 on dist-test-slave-b69v"
server_key: "1e6f38e1d0a570b817b238c674bc8804"
server_key_iv: "7b69478d9e74e5ebb3b0efda2e5f0676"
server_key_version: "encryptionkey@0"
I20260929 16:03:37.893573 27690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:37.912031 27690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:37.915844 27690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:37.916167 27690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:37.935760 27690 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:35471
I20260929 16:03:37.935786 27756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:35471 every 8 connection(s)
I20260929 16:03:37.937290 27690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:37.941430 27757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:37.944593 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27690
I20260929 16:03:37.944794 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:37.945364 5588 external_mini_cluster.cc:1428] Setting key 344512cbfa8f5a923d9812ec5e96a22e
I20260929 16:03:37.954783 27757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d: Bootstrap starting.
I20260929 16:03:37.959476 27757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:37.961045 27757 log.cc:824] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:03:37 dist-test-slave-b69v krb5kdc[27677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:37.965291 27757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d: No bootstrap required, opened a new log
I20260929 16:03:37.973479 27760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:37.947386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40321 (local address 127.5.117.62:35471)
0929 16:03:37.948246 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:37.948258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:37.948303 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:03:37.949975 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:37.949997 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:37.950424 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:37.950780 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:37.952818 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:37.954233 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:37.956154 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:37.956920 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:37.957074 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:37.962900 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:37.962912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:37.962937 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:37.962986 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:37.967741 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:37.968783 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:37.968793 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:37.968803 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:37.969040 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:37.969536 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:37.969541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:37.969545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:37.970422 (+ 877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:37.970774 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:37.971231 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 16:03:37.971793 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":392,"threads_started":1}
I20260929 16:03:37.976979 27757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } }
I20260929 16:03:37.977469 27757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:37.977608 27757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ccf08cacca0477babf708de41acf39d, State: Initialized, Role: FOLLOWER
I20260929 16:03:37.978297 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:37.978317 27757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } }
I20260929 16:03:37.978602 27757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:37.978730 27757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:37.978891 27757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:37.982376 27757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } }
I20260929 16:03:37.983098 27757 leader_election.cc:302] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ccf08cacca0477babf708de41acf39d; no voters:
I20260929 16:03:37.983959 27757 leader_election.cc:288] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:37.984323 27766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:37.985550 27766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [term 1 LEADER]: Becoming Leader. State: Replica: 0ccf08cacca0477babf708de41acf39d, State: Running, Role: LEADER
I20260929 16:03:37.986728 27757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:37.986691 27766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } }
I20260929 16:03:37.991612 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ccf08cacca0477babf708de41acf39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } } }
I20260929 16:03:37.992035 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:37.991984 27769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ccf08cacca0477babf708de41acf39d. Latest consensus state: current_term: 1 leader_uuid: "0ccf08cacca0477babf708de41acf39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccf08cacca0477babf708de41acf39d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 35471 } } }
I20260929 16:03:37.992340 27769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:38.000025 27773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:38.006968 27760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:37.984767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35909 (local address 127.5.117.62:35471)
0929 16:03:37.984980 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:37.984985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:37.987427 (+ 2442us) server_negotiation.cc:407] Connection header received
0929 16:03:37.987624 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:37.987629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:37.987713 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:37.987820 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:37.990262 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:37.991611 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:37.994099 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:37.995138 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:37.995309 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:37.998714 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:37.998725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:37.998732 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:37.998783 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:38.003300 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:38.004247 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:38.004252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:38.004256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:38.004401 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:38.004929 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:38.004938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:38.004941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:38.005601 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:38.005968 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:38.006269 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 16:03:38.006535 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:03:38.007409 27773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:38.018756 27773 catalog_manager.cc:1347] Generated new cluster ID: 5763dc9e641243f6b123de0695c9a563
I20260929 16:03:38.018903 27773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:38.028734 27773 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:38.030651 27773 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:38.040823 27773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ccf08cacca0477babf708de41acf39d: Generated new TSK 0
I20260929 16:03:38.041929 27773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:38.048846 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27690
I20260929 16:03:38.050011 27752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:38.149963 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27690
2026-09-29T16:03:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:38 dist-test-slave-b69v krb5kdc[27789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:38 dist-test-slave-b69v krb5kdc[27789](info): set up 2 sockets
Sep 29 16:03:38 dist-test-slave-b69v krb5kdc[27789](info): commencing operation
krb5kdc: starting...
W20260929 16:03:39.252959 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.008s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:39 dist-test-slave-b69v krb5kdc[27789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:39.589437 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38365
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:44079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:38365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:39.978106 27802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:39.978493 27802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:39.978600 27802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:39.987512 27802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:39.987641 27802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:39.987694 27802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:39.987759 27802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:39.987807 27802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:39.999003 27802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:44079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:38365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:40.000618 27802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:40.002287 27802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:40.015704 27807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:40.015739 27810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:40.015797 27808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:40.017340 27802 server_base.cc:1034] running on GCE node
I20260929 16:03:40.018160 27802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:40.019685 27802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:40.020927 27802 hybrid_clock.cc:648] HybridClock initialized: now 1790697820020919 us; error 119 us; skew 500 ppm
I20260929 16:03:40.021312 27802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:40 dist-test-slave-b69v krb5kdc[27789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697820, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:40.030247 27802 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:40.032146 27802 webserver.cc:460] Webserver started at http://127.5.117.62:36949/ using document root <none> and password file <none>
I20260929 16:03:40.032989 27802 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:40.033154 27802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:40.033509 27802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:40.037118 27802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2"
format_stamp: "Formatted at 2026-09-29 16:03:40 on dist-test-slave-b69v"
server_key: "bbdd02a42a77f9f8fb72b2dc54d6ffe8"
server_key_iv: "82a02e6faf0e254fccd088c4112e694e"
server_key_version: "encryptionkey@0"
I20260929 16:03:40.038074 27802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2"
format_stamp: "Formatted at 2026-09-29 16:03:40 on dist-test-slave-b69v"
server_key: "bbdd02a42a77f9f8fb72b2dc54d6ffe8"
server_key_iv: "82a02e6faf0e254fccd088c4112e694e"
server_key_version: "encryptionkey@0"
I20260929 16:03:40.044363 27802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:03:40.048406 27817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:40.049945 27802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:40.050144 27802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2"
format_stamp: "Formatted at 2026-09-29 16:03:40 on dist-test-slave-b69v"
server_key: "bbdd02a42a77f9f8fb72b2dc54d6ffe8"
server_key_iv: "82a02e6faf0e254fccd088c4112e694e"
server_key_version: "encryptionkey@0"
I20260929 16:03:40.050376 27802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:40.072865 27802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:40.077594 27802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:40.077930 27802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:40.099651 27802 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:38365
I20260929 16:03:40.099676 27868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:38365 every 8 connection(s)
I20260929 16:03:40.101867 27802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:40.106310 27869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:40.107561 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27802
I20260929 16:03:40.107777 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:40.108381 5588 external_mini_cluster.cc:1428] Setting key 91f7288e005dd3d2d15898f67efcd5c2
I20260929 16:03:40.121387 27869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2: Bootstrap starting.
I20260929 16:03:40.126394 27869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:40.127038 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:40.110776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54315 (local address 127.5.117.62:38365)
0929 16:03:40.111944 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:40.111972 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:40.112382 (+ 410us) server_negotiation.cc:407] Connection header received
0929 16:03:40.115064 (+ 2682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:40.115101 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:40.115654 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:40.116169 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:40.116759 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:40.118958 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:40.120439 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:40.121525 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:40.121745 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:40.123073 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:40.123087 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:40.123105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:40.124022 (+ 917us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:40.124174 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:40.124213 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:40.124362 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:40.124847 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 16:03:40.125202 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":498,"threads_started":1}
I20260929 16:03:40.127992 27869 log.cc:824] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:40.131847 27869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2: No bootstrap required, opened a new log
I20260929 16:03:40.132470 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:40.142057 27869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } }
I20260929 16:03:40.142681 27869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:40.142802 27869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed9eb0319f94c97b2800d87cb0ad3b2, State: Initialized, Role: FOLLOWER
I20260929 16:03:40.143571 27869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } }
I20260929 16:03:40.143874 27869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:40.144068 27869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:40.144292 27869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:40.144762 27872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:40.137413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36175 (local address 127.5.117.62:38365)
0929 16:03:40.137606 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:40.137611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:40.138152 (+ 541us) server_negotiation.cc:407] Connection header received
0929 16:03:40.138324 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:40.138331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:40.138427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:40.138548 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:40.139017 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:40.140258 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:40.141673 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:40.142373 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:40.142520 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:40.143865 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:40.143874 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:40.143880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:40.143906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:40.143958 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:40.143983 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:40.144144 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:40.144287 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:03:40.144398 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:03:40.147704 27869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } }
I20260929 16:03:40.148394 27869 leader_election.cc:302] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ed9eb0319f94c97b2800d87cb0ad3b2; no voters:
I20260929 16:03:40.149024 27869 leader_election.cc:288] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:40.149211 27879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:40.150475 27879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed9eb0319f94c97b2800d87cb0ad3b2, State: Running, Role: LEADER
I20260929 16:03:40.151095 27879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } }
I20260929 16:03:40.151791 27869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:40.154261 27880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } } }
I20260929 16:03:40.154731 27880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:40.154281 27881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ed9eb0319f94c97b2800d87cb0ad3b2. Latest consensus state: current_term: 1 leader_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ed9eb0319f94c97b2800d87cb0ad3b2" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 38365 } } }
I20260929 16:03:40.155848 27881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:40.156769 27886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:40.162464 27886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:40.171614 27886 catalog_manager.cc:1347] Generated new cluster ID: dae0823f0ed94ecf9ed779269439c461
I20260929 16:03:40.171736 27886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:40.183983 27886 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:40.186012 27886 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:40.197005 27886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ed9eb0319f94c97b2800d87cb0ad3b2: Generated new TSK 0
I20260929 16:03:40.198627 27886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:40.206235 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27802
W20260929 16:03:40.207424 27864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54315
W20260929 16:03:40.208271 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27802: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:40.208429 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27802
2026-09-29T16:03:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2058 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:40 dist-test-slave-b69v krb5kdc[27900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:40 dist-test-slave-b69v krb5kdc[27900](info): set up 2 sockets
Sep 29 16:03:40 dist-test-slave-b69v krb5kdc[27900](info): commencing operation
krb5kdc: starting...
W20260929 16:03:41.353108 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:41 dist-test-slave-b69v krb5kdc[27900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:41.704150 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41411
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:42.104354 27913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:42.104741 27913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:42.104921 27913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:42.112759 27913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:42.112937 27913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:42.113056 27913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:42.113173 27913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:42.113258 27913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:42.123932 27913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:42.125598 27913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:42.127198 27913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:42.141132 27918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:42.141271 27921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:42.141353 27919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:42.142482 27913 server_base.cc:1034] running on GCE node
I20260929 16:03:42.143414 27913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:42.145182 27913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:42.146441 27913 hybrid_clock.cc:648] HybridClock initialized: now 1790697822146388 us; error 39 us; skew 500 ppm
I20260929 16:03:42.146989 27913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:42 dist-test-slave-b69v krb5kdc[27900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697822, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:42.151079 27913 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:42.153156 27913 webserver.cc:460] Webserver started at http://127.5.117.62:34435/ using document root <none> and password file <none>
I20260929 16:03:42.153997 27913 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:42.154165 27913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:42.154536 27913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:42.158903 27913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "0447d363ca964fba84e453ca79e4d45d"
format_stamp: "Formatted at 2026-09-29 16:03:42 on dist-test-slave-b69v"
server_key: "2d1f6688d11de50e3634e397c606473f"
server_key_iv: "0195743cde08899a7e1f8afd78773fea"
server_key_version: "encryptionkey@0"
I20260929 16:03:42.160144 27913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "0447d363ca964fba84e453ca79e4d45d"
format_stamp: "Formatted at 2026-09-29 16:03:42 on dist-test-slave-b69v"
server_key: "2d1f6688d11de50e3634e397c606473f"
server_key_iv: "0195743cde08899a7e1f8afd78773fea"
server_key_version: "encryptionkey@0"
I20260929 16:03:42.168645 27913 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 16:03:42.172675 27928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:42.174312 27913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:03:42.174546 27913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "0447d363ca964fba84e453ca79e4d45d"
format_stamp: "Formatted at 2026-09-29 16:03:42 on dist-test-slave-b69v"
server_key: "2d1f6688d11de50e3634e397c606473f"
server_key_iv: "0195743cde08899a7e1f8afd78773fea"
server_key_version: "encryptionkey@0"
I20260929 16:03:42.174793 27913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:42.181609 27913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:42.186118 27913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:42.186431 27913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:42.207335 27913 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:41411
I20260929 16:03:42.207373 27979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:41411 every 8 connection(s)
I20260929 16:03:42.209121 27913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:42.211457 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 27913
I20260929 16:03:42.211675 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:42.212262 5588 external_mini_cluster.cc:1428] Setting key 07354ca2fb37cf241c1ec9bdec2c6d15
I20260929 16:03:42.214344 27980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:42.221504 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:42.214476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58115 (local address 127.5.117.62:41411)
0929 16:03:42.215377 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:42.215390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:42.215436 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:03:42.217320 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:42.217345 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:42.217744 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:42.218110 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:42.219335 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:42.219342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:42.219355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:42.219380 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:42.219471 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:42.219493 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:42.219684 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:42.219794 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:03:42.220016 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":411,"threads_started":1}
I20260929 16:03:42.226999 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:42.229393 27980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d: Bootstrap starting.
I20260929 16:03:42.233968 27980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:42.235611 27980 log.cc:824] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:42.236006 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:42.232126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39219 (local address 127.5.117.62:41411)
0929 16:03:42.232388 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:42.232398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:42.233037 (+ 639us) server_negotiation.cc:407] Connection header received
0929 16:03:42.233257 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:42.233262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:42.233323 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:42.233414 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:42.235016 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:42.235024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:42.235028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:42.235071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:42.235153 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:42.235183 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:42.235359 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:42.235568 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 16:03:42.235635 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:42.239280 27980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d: No bootstrap required, opened a new log
I20260929 16:03:42.247577 27980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } }
I20260929 16:03:42.248021 27980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:42.248174 27980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0447d363ca964fba84e453ca79e4d45d, State: Initialized, Role: FOLLOWER
I20260929 16:03:42.248829 27980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } }
I20260929 16:03:42.249094 27980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:42.249275 27980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:42.249509 27980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:42.251972 27980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } }
I20260929 16:03:42.252621 27980 leader_election.cc:302] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0447d363ca964fba84e453ca79e4d45d; no voters:
I20260929 16:03:42.253405 27980 leader_election.cc:288] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:42.253738 27990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:42.254652 27990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [term 1 LEADER]: Becoming Leader. State: Replica: 0447d363ca964fba84e453ca79e4d45d, State: Running, Role: LEADER
I20260929 16:03:42.255379 27990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } }
I20260929 16:03:42.256287 27980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:42.258641 27991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0447d363ca964fba84e453ca79e4d45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } } }
I20260929 16:03:42.259183 27991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:42.258615 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0447d363ca964fba84e453ca79e4d45d. Latest consensus state: current_term: 1 leader_uuid: "0447d363ca964fba84e453ca79e4d45d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0447d363ca964fba84e453ca79e4d45d" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 41411 } } }
I20260929 16:03:42.259788 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:42.260866 27998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:42.266882 27998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:42.277330 27998 catalog_manager.cc:1347] Generated new cluster ID: f0eefb558f3446ba8968c544ad231d9e
I20260929 16:03:42.277441 27998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:42.308364 27998 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:42.308995 27998 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:42.322654 27998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0447d363ca964fba84e453ca79e4d45d: Generated new TSK 0
I20260929 16:03:42.323882 27998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:42.338243 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27913
W20260929 16:03:42.339746 27975 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58115
W20260929 16:03:42.340536 5588 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 27913: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:42.340699 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 27913
2026-09-29T16:03:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2133 ms)
[----------] 18 tests from AuthTokenIssuingTest (37267 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:42 dist-test-slave-b69v krb5kdc[28011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:42 dist-test-slave-b69v krb5kdc[28011](info): set up 2 sockets
Sep 29 16:03:42 dist-test-slave-b69v krb5kdc[28011](info): commencing operation
krb5kdc: starting...
W20260929 16:03:43.477415 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.006s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:43 dist-test-slave-b69v krb5kdc[28011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:43.832399 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46867
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:44.221328 28024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:44.221758 28024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:44.221896 28024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:44.222059 28024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:44.229732 28024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:44.229832 28024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:44.229895 28024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:44.229990 28024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:44.230109 28024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:44.240706 28024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:46867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:44.242290 28024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:44.243865 28024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:44.257426 28029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:44.257397 28030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:44.257530 28032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:44.258137 28024 server_base.cc:1034] running on GCE node
I20260929 16:03:44.258955 28024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:44.260506 28024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:44.261759 28024 hybrid_clock.cc:648] HybridClock initialized: now 1790697824261706 us; error 44 us; skew 500 ppm
I20260929 16:03:44.262156 28024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:44 dist-test-slave-b69v krb5kdc[28011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:44.265697 28024 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:44.267361 28024 webserver.cc:460] Webserver started at http://127.5.117.62:41687/ using document root <none> and password file <none>
I20260929 16:03:44.268252 28024 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:44.268394 28024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:44.268769 28024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:44.272435 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "fd7b45a978df4c38b756394cba2dfa35"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "a5f45704bb3af287e798cf7e42e89e30"
server_key_iv: "5927cffb1312e98aa92d3c5a029dafe9"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.273366 28024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "fd7b45a978df4c38b756394cba2dfa35"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "a5f45704bb3af287e798cf7e42e89e30"
server_key_iv: "5927cffb1312e98aa92d3c5a029dafe9"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.280582 28024 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 16:03:44.284677 28039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:44.286186 28024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:03:44.286396 28024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "fd7b45a978df4c38b756394cba2dfa35"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "a5f45704bb3af287e798cf7e42e89e30"
server_key_iv: "5927cffb1312e98aa92d3c5a029dafe9"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.286675 28024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:44.309382 28024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:44.314028 28024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:44.314332 28024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:44.337644 28024 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:46867
I20260929 16:03:44.337656 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:46867 every 8 connection(s)
I20260929 16:03:44.339267 28024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:44.343750 28091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:44.346089 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28024
I20260929 16:03:44.346357 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:44.347260 5588 external_mini_cluster.cc:1428] Setting key 8fde7d2e9110d8adcdb2e55468c2b41a
I20260929 16:03:44.355232 28091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:44 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:44.382114 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.359878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.5.117.62:46867)
0929 16:03:44.361835 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:44.361847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:44.361906 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:03:44.363588 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:44.363593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:44.363824 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:44.364053 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:44.365502 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.366791 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.368660 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.369219 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.369425 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.374239 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:44.374247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:44.374259 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:44.374296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:44.377278 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.378329 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.378336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.378338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.378497 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.379030 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.379034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.379037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.379609 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:44.379790 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:44.380575 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 16:03:44.380910 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":1724,"thread_start_us":1467,"threads_started":1}
I20260929 16:03:44.382164 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.359877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46867 (local address 127.0.0.1:41728)
0929 16:03:44.360974 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.361043 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:44.363189 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:44.364086 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:44.364093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:44.364903 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:44.365269 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.365286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.366965 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.366974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.368510 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.368524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.369470 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.369477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.369693 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.370536 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:44.370578 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:44.373951 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:44.377453 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.377469 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.377483 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.378164 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.378637 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.378645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.378652 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.378877 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.379768 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:44.379779 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:44.380080 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 16:03:44.380380 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 16:03:44.381100 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":162,"thread_start_us":538,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.61@KRBTEST.COM; defaulting to no policy
I20260929 16:03:44.385572 28091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:44.386847 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.386357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54338)
0929 16:03:44.386602 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.386732 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 16:03:44.387121 28040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.61@KRBTEST.COM" created.
I20260929 16:03:44.388705 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.388384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54342)
0929 16:03:44.388518 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.388585 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:03:44.388917 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:44215: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
I20260929 16:03:44.429682 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:44215
--webserver_interface=127.5.117.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:44.438270 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } attempt: 1
I20260929 16:03:44.439666 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.439134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54354)
0929 16:03:44.439369 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.439506 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:03:44.440850 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.440460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54366)
0929 16:03:44.440634 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.440711 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 16:03:44.441051 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:44215: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
I20260929 16:03:44.509352 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } attempt: 2
I20260929 16:03:44.511046 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.510394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54370)
0929 16:03:44.510705 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.510836 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:03:44.512485 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.511878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54374)
0929 16:03:44.512132 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.512251 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 16:03:44.512737 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:44215: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
I20260929 16:03:44.604074 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } attempt: 3
I20260929 16:03:44.605651 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.605046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54388)
0929 16:03:44.605306 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.605439 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:03:44.606829 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.606316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54398)
0929 16:03:44.606527 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.606639 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 16:03:44.607124 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:44215: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
I20260929 16:03:44.750481 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } attempt: 4
I20260929 16:03:44.752230 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.751408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54410)
0929 16:03:44.751825 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.751956 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260929 16:03:44.753854 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.753327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54424)
0929 16:03:44.753551 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.753679 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 16:03:44.754117 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:44215: Network error: Client connection negotiation failed: client connection to 127.5.117.61:44215: connect: Connection refused (error 111)
W20260929 16:03:44.815791 28098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:44.816116 28098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:44.816247 28098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:44.816365 28098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:44.823081 28098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:44.823159 28098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:44.823277 28098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:44.823366 28098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:44.823506 28098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:44.832739 28098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.117.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:44.834141 28098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:44.835587 28098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:44.849052 28103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:44.849128 28104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:44.849159 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:44.850476 28098 server_base.cc:1034] running on GCE node
I20260929 16:03:44.851505 28098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:44.853565 28098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:44.854883 28098 hybrid_clock.cc:648] HybridClock initialized: now 1790697824854772 us; error 69 us; skew 500 ppm
I20260929 16:03:44.855325 28098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:44 dist-test-slave-b69v krb5kdc[28011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:44.859303 28098 init.cc:377] Logged in from keytab as kudu/127.5.117.61@KRBTEST.COM (short username kudu)
I20260929 16:03:44.861342 28098 webserver.cc:460] Webserver started at http://127.5.117.61:45317/ using document root <none> and password file <none>
I20260929 16:03:44.862257 28098 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:44.862403 28098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:44.862803 28098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:44.866606 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data/instance:
uuid: "8e6c2dd2386d4c2dbce9c2caa6096239"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "109fe41797c8141316f76629c8bd5722"
server_key_iv: "0bde42f32554f150afe0afc00b87e550"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.867574 28098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal/instance:
uuid: "8e6c2dd2386d4c2dbce9c2caa6096239"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "109fe41797c8141316f76629c8bd5722"
server_key_iv: "0bde42f32554f150afe0afc00b87e550"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.874142 28098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260929 16:03:44.879240 28113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:44.881210 28098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:03:44.881526 28098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "8e6c2dd2386d4c2dbce9c2caa6096239"
format_stamp: "Formatted at 2026-09-29 16:03:44 on dist-test-slave-b69v"
server_key: "109fe41797c8141316f76629c8bd5722"
server_key_iv: "0bde42f32554f150afe0afc00b87e550"
server_key_version: "encryptionkey@0"
I20260929 16:03:44.881996 28098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:44.903512 28098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:44.908114 28098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:44.908425 28098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:44.929838 28098 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.61:44215
I20260929 16:03:44.929859 28164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.61:44215 every 8 connection(s)
I20260929 16:03:44.931348 28098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
I20260929 16:03:44.933531 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28098
I20260929 16:03:44.933758 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-1/wal/instance
I20260929 16:03:44.934615 5588 external_mini_cluster.cc:1428] Setting key 3ab5ce3dbde23e393cdd4c03e2977d08
I20260929 16:03:44.936754 28165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:44.945613 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:44 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:44.969189 28168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.948681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.5.117.62:46867)
0929 16:03:44.949267 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:44.949275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:44.950210 (+ 935us) server_negotiation.cc:407] Connection header received
0929 16:03:44.952275 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:44.952284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:44.952413 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:44.952555 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:44.954128 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.955323 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.956942 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.957706 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.957860 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.961494 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:44.961504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:44.961512 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:44.961578 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:44.965580 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.966393 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.966401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.966406 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.966556 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.967145 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.967148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.967150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.967450 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:44.967565 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:44.968676 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0929 16:03:44.968872 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260929 16:03:44.970758 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.949235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46867 (local address 127.0.0.1:41744)
0929 16:03:44.950036 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.950076 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:44.951937 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:44.952753 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:44.952761 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:44.953297 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:44.953896 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.953913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.955485 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.955489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.956790 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.956802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.957928 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.957933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.958135 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.958901 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:44.958925 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:44.961248 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:44.965794 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.965801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.965814 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.966249 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.966718 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.966725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.966731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.967031 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.967582 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:44.967593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:44.968232 (+ 639us) client_negotiation.cc:769] Sending connection context
0929 16:03:44.968587 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 16:03:44.969222 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.60@KRBTEST.COM; defaulting to no policy
I20260929 16:03:44.972744 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.60@KRBTEST.COM" created.
Sep 29 16:03:44 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:44.988356 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.973652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54438)
0929 16:03:44.973818 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.973840 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:44.973943 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:44.975176 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:44.975183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:44.975741 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:44.975965 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.975982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.977350 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.977357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.978688 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:44.978704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.979488 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.979495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:44.979707 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.980387 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:44.980414 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:44.982871 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:44.985888 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.985894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.985900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.986216 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.986578 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:44.986584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:44.986606 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:44.986797 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:44.987389 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:44.987399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:44.987588 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 16:03:44.987724 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:03:44.987983 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:03:44.988516 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.973938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.5.117.61:44215)
0929 16:03:44.974699 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:44.974705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:44.974731 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:03:44.974771 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:44.974778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:44.974987 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:44.975108 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:44.976200 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.977202 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.978825 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:44.979337 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:44.979430 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:44.983090 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:44.983101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:44.983116 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:44.983143 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:44.985762 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.986333 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.986361 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.986366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.986482 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:44.986925 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:44.986929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:44.986932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:44.987271 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:44.987486 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:44.987922 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:03:44.988190 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":458,"threads_started":1}
I20260929 16:03:44.990866 28165 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:44.992197 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.991666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47000)
0929 16:03:44.991839 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.991982 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:03:44.992564 28115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:44.994323 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:44.994026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47006)
0929 16:03:44.994151 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:44.994227 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:03:44.994643 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
I20260929 16:03:45.015232 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:37377
--webserver_interface=127.5.117.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:45.043958 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 1
I20260929 16:03:45.045358 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.044863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47014)
0929 16:03:45.045093 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.045198 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:03:45.046301 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.045984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47016)
0929 16:03:45.046134 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.046201 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:03:45.046494 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.053357 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } attempt: 5
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:45.075713 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.054013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54442)
0929 16:03:45.054244 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.054263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.054444 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.054866 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.054871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.055417 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.055679 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.055692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.056890 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.056897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.058309 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.058328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.059124 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.059130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.059368 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.066382 (+ 7014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.066420 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.069484 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.073046 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.073052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.073056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.073468 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.073862 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.073867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.073871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.074030 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.074604 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.074609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.074794 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.074985 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.075317 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:03:45.075716 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.054081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.5.117.61:44215)
0929 16:03:45.054252 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.054257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.054363 (+ 106us) server_negotiation.cc:407] Connection header received
0929 16:03:45.054622 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.054626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.054711 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.054804 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.055845 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.056710 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.058471 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.058963 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.059067 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.069766 (+ 10699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.069772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.069775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.069819 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.072880 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.073595 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.073603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.073606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.073741 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.074144 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.074150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.074154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.074479 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.074644 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.075074 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.075325 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:03:45.077093 28091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:45.078317 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.077900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47026)
0929 16:03:45.078090 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.078178 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:03:45.079306 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.078995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47032)
0929 16:03:45.079130 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.079218 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:03:45.079543 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.114727 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 2
I20260929 16:03:45.115931 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.115479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47044)
0929 16:03:45.115674 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.115765 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:03:45.116935 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.116588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47054)
0929 16:03:45.116750 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.116818 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:03:45.117149 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.130777 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 1
I20260929 16:03:45.132225 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.131482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47062)
0929 16:03:45.131688 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.131770 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:03:45.133368 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.133026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47070)
0929 16:03:45.133187 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.133256 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:03:45.133616 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.201835 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 2
I20260929 16:03:45.203293 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.202745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47084)
0929 16:03:45.203006 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.203106 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:03:45.204310 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.203959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47098)
0929 16:03:45.204143 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.204210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:03:45.204505 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.208359 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 3
I20260929 16:03:45.209338 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.208900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47104)
0929 16:03:45.209103 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.209192 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:03:45.210342 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.209923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47114)
0929 16:03:45.210161 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.210233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260929 16:03:45.210546 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.310755 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 3
I20260929 16:03:45.312660 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.311863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47122)
0929 16:03:45.312320 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.312456 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260929 16:03:45.314193 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.313701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47128)
0929 16:03:45.313950 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.314050 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260929 16:03:45.314421 28091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
I20260929 16:03:45.353770 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 4
I20260929 16:03:45.355122 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.354568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47140)
0929 16:03:45.354839 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.354934 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:03:45.356242 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.355808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47156)
0929 16:03:45.355982 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.356097 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:03:45.356446 28165 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:37377: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
W20260929 16:03:45.389847 28173 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:45.390383 28173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:45.390525 28173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:45.390679 28173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:45.398438 28173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:45.398557 28173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:45.398693 28173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:45.398787 28173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:45.398904 28173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:45.409240 28173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:46867,127.5.117.61:44215,127.5.117.60:37377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.117.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:45.410727 28173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:45.412256 28173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:45.425959 28178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:45.425920 28179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:45.426028 28181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:45.426841 28173 server_base.cc:1034] running on GCE node
I20260929 16:03:45.427706 28173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:45.429207 28173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:45.430460 28173 hybrid_clock.cc:648] HybridClock initialized: now 1790697825430394 us; error 41 us; skew 500 ppm
I20260929 16:03:45.430811 28173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:45.434332 28173 init.cc:377] Logged in from keytab as kudu/127.5.117.60@KRBTEST.COM (short username kudu)
I20260929 16:03:45.436159 28173 webserver.cc:460] Webserver started at http://127.5.117.60:40347/ using document root <none> and password file <none>
I20260929 16:03:45.436913 28173 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:45.437062 28173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:45.437412 28173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:45.440686 28173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data/instance:
uuid: "498d01c81d4449aeabfc32700b854835"
format_stamp: "Formatted at 2026-09-29 16:03:45 on dist-test-slave-b69v"
server_key: "d3c014faa430378b6655a8f49b6407c0"
server_key_iv: "dc3a28a8fb58250ae11d4b6e4b7134ef"
server_key_version: "encryptionkey@0"
I20260929 16:03:45.441532 28173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal/instance:
uuid: "498d01c81d4449aeabfc32700b854835"
format_stamp: "Formatted at 2026-09-29 16:03:45 on dist-test-slave-b69v"
server_key: "d3c014faa430378b6655a8f49b6407c0"
server_key_iv: "dc3a28a8fb58250ae11d4b6e4b7134ef"
server_key_version: "encryptionkey@0"
I20260929 16:03:45.447038 28173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:03:45.450663 28188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:45.452143 28173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:03:45.452347 28173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "498d01c81d4449aeabfc32700b854835"
format_stamp: "Formatted at 2026-09-29 16:03:45 on dist-test-slave-b69v"
server_key: "d3c014faa430378b6655a8f49b6407c0"
server_key_iv: "dc3a28a8fb58250ae11d4b6e4b7134ef"
server_key_version: "encryptionkey@0"
I20260929 16:03:45.452579 28173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:45.465989 28173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:45.470662 28173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:45.470966 28173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:45.491669 28091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 4
I20260929 16:03:45.493212 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.492579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47162)
0929 16:03:45.492901 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.493009 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 16:03:45.493255 28173 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.60:37377
I20260929 16:03:45.493284 28239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.60:37377 every 8 connection(s)
I20260929 16:03:45.495187 28173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
I20260929 16:03:45.496615 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28173
I20260929 16:03:45.496901 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697441705944-5588-0/minicluster-data/master-2/wal/instance
I20260929 16:03:45.497723 5588 external_mini_cluster.cc:1428] Setting key f9ea3ed08e1a1da14c7f82deb14e2dea
I20260929 16:03:45.501540 28240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:45.517349 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:45.533573 28093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.493980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47168)
0929 16:03:45.494171 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.494195 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.494344 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.512472 (+ 18128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.512478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.513144 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.513408 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.513426 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.516003 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.516012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.517330 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.517364 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.519610 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.519620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.519846 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.520964 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.520996 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.524728 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.530220 (+ 5492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.530227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.530231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.530803 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.531349 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.531355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.531359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.531511 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.532514 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.532519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.532692 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.532846 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.533197 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:45.535151 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.495383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.5.117.60:37377)
0929 16:03:45.509159 (+ 13776us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.509182 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.509258 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:03:45.511593 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.511616 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.512032 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.512438 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.513599 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.515795 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.517690 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.519438 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.520287 (+ 849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.525075 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.525085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.525102 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.525153 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.530037 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.530984 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.530995 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.531005 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.531213 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.531637 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.531643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.531646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.532368 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.532585 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.533168 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.533648 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":815,"server-negotiator.queue_time_us":13453,"spinlock_wait_cycles":38784,"thread_start_us":13188,"threads_started":1}
I20260929 16:03:45.535179 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.509000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.5.117.60:37377)
0929 16:03:45.510416 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.510420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.510439 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:03:45.511611 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.511617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.512032 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.512440 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.514399 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.515808 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.517742 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.519426 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.519669 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.526293 (+ 6624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.526302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.526310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.526361 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.530523 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.531420 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.531425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.531428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.531614 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.532033 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.532039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.532042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.532626 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.533153 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.534537 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.534746 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":1311,"spinlock_wait_cycles":39424,"thread_start_us":1165,"threads_started":1}
I20260929 16:03:45.543810 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: Bootstrap starting.
I20260929 16:03:45.547513 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.520228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:46867 (local address 127.0.0.1:41754)
0929 16:03:45.523830 (+ 3602us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.523888 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.524071 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.525551 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.525561 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.526437 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.526759 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.526768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.528205 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.528208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.529590 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.529605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.530406 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.530413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.530633 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.531309 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.531341 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.534721 (+ 3380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.544368 (+ 9647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.544381 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.544395 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.544820 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.545231 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.545236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.545247 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.545386 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.546211 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.546219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.546441 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.546668 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.547020 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3390,"thread_start_us":3951,"threads_started":1}
I20260929 16:03:45.548590 28245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.520135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.5.117.62:46867)
0929 16:03:45.525166 (+ 5031us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.525172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.525192 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:03:45.525251 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.525259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.525339 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.525459 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.526920 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.527962 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.529768 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.530200 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.530360 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.534934 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.534940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.534945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.534990 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.544239 (+ 9249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.544950 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.544957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.544971 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.545102 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.545540 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.545545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.545548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.546052 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.546246 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.547845 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.548136 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4890,"thread_start_us":4747,"threads_started":1}
I20260929 16:03:45.548777 28091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:45.550354 28091 log.cc:824] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:45.550350 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:45.555954 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: No bootstrap required, opened a new log
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:45.560618 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.540101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.5.117.62:46867)
0929 16:03:45.541100 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.541106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.541142 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:03:45.541290 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.541295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.541379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.541501 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.543050 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.543934 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.545691 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.546427 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.546568 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.552416 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.552428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.552435 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.552492 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.556527 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.557434 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.557441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.557447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.557637 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.558162 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.558167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.558170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.558635 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.559844 (+ 1209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.559946 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.560211 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":370,"threads_started":1}
I20260929 16:03:45.567526 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.551185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:44215 (local address 127.0.0.1:54454)
0929 16:03:45.551419 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.551439 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.551595 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.552320 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.552346 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.552822 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.552967 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.552977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.554238 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.554244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.555332 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.555344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.556204 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.556212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.556401 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.557180 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.557206 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.560363 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.564687 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.564692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.564696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.565217 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.565720 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.565725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.565730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.565901 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.566519 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.566524 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.566698 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.566873 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.567175 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:03:45.567652 28091 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.568104 28091 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7b45a978df4c38b756394cba2dfa35, State: Initialized, Role: FOLLOWER
I20260929 16:03:45.568560 28170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.551209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.5.117.61:44215)
0929 16:03:45.551424 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.551434 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.551721 (+ 287us) server_negotiation.cc:407] Connection header received
0929 16:03:45.551815 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.551823 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.551935 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.552073 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.553134 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.554065 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.555486 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.555937 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.556136 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.560597 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.560607 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.560613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.560666 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.564525 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.565365 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.565374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.565377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.565581 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.566040 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.566045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.566049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.566367 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.566552 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.567843 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.568131 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:03:45.568715 28091 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.569721 28250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.569981 28240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:45.570135 28250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.571341 28091 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:45.580526 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.563032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.5.117.61:44215)
0929 16:03:45.563707 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.563712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.563732 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:03:45.563770 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.563773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.563827 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.563964 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.565317 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.566534 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.567957 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.568735 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.572374 (+ 3639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.574170 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.574179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.574186 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.574224 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.577616 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.578279 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.578286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.578289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.578422 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.578797 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.578804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.578807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.579083 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.579358 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.579618 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.579849 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":334,"threads_started":1}
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:45.586647 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:45.588721 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.570942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47188)
0929 16:03:45.571289 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.571309 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.571416 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.571842 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.571847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.572374 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.572576 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.572590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.574002 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.574010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.575740 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.575760 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.577004 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.577009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.577243 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.577907 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.577946 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.581585 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.585698 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.585703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.585708 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.586226 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.586709 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.586715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.586719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.586866 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.587678 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.587683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.587835 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.587999 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.588337 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
I20260929 16:03:45.589386 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.571310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.5.117.60:37377)
0929 16:03:45.571534 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.571541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.571562 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:03:45.571614 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.571620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.571692 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.571793 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.572751 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.573851 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.576164 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.576839 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.576976 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.582252 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.582257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.582260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.582287 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.585555 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.586360 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.586364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.586368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.586581 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.586991 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.586999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.587005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.587462 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.587667 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.588153 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.588558 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":68}
I20260929 16:03:45.596539 28066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd7b45a978df4c38b756394cba2dfa35"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41762
I20260929 16:03:45.597239 28066 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:03:45.597517 28066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:45.598191 28240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: Bootstrap starting.
I20260929 16:03:45.602216 28066 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.602775 28240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: Neither blocks nor log segments found. Creating new log.
W20260929 16:03:45.602979 28261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:45.603155 28261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:45.589411 28024 master_runner.cc:186] Error getting master registration for 127.5.117.61:44215: OK, instance_id { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" instance_seqno: 1790697824924360 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:45.604852 28240 log.cc:824] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:45.605558 28066 leader_election.cc:288] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [CANDIDATE]: Term 1 election: Requested vote from peers 8e6c2dd2386d4c2dbce9c2caa6096239 (127.5.117.61:44215), 498d01c81d4449aeabfc32700b854835 (127.5.117.60:37377)
I20260929 16:03:45.605859 28024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:45.624142 28240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: No bootstrap required, opened a new log
I20260929 16:03:45.636323 28240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.637076 28240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498d01c81d4449aeabfc32700b854835, State: Initialized, Role: FOLLOWER
I20260929 16:03:45.637795 28240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.639568 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.640599 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.641039 28240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:45.641965 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd7b45a978df4c38b756394cba2dfa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "498d01c81d4449aeabfc32700b854835"
I20260929 16:03:45.642594 28215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:45.646409 28215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7b45a978df4c38b756394cba2dfa35 in term 1.
I20260929 16:03:45.647140 28041 leader_election.cc:302] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498d01c81d4449aeabfc32700b854835, fd7b45a978df4c38b756394cba2dfa35; no voters:
I20260929 16:03:45.647760 28250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:45.651495 28250 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [term 1 LEADER]: Becoming Leader. State: Replica: fd7b45a978df4c38b756394cba2dfa35, State: Running, Role: LEADER
I20260929 16:03:45.652175 28250 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.653244 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.610135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.5.117.62:46867)
0929 16:03:45.610429 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.610440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.618606 (+ 8166us) server_negotiation.cc:407] Connection header received
0929 16:03:45.621565 (+ 2959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.621574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.621673 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.621764 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.632183 (+ 10419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.633326 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.635429 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.637101 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.637230 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.642277 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.642284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.642290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.642333 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.646761 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.648310 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.648344 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.648348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.648623 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.649620 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.649625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.649629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.650439 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.650913 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.652106 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.652596 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:03:45.655671 28165 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } attempt: 5
I20260929 16:03:45.657611 28273 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7b45a978df4c38b756394cba2dfa35. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.658252 28273 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:45.660594 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.610736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54470 (local address 127.5.117.61:44215)
0929 16:03:45.611040 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.611049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.617051 (+ 6002us) server_negotiation.cc:407] Connection header received
0929 16:03:45.617478 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.617488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.617617 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.617719 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.625242 (+ 7523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.626850 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.634733 (+ 7883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.635727 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.635878 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.640976 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.640983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.640988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.641038 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.651293 (+ 10255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.657850 (+ 6557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.657856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.657860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.658026 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.658531 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.658537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.658541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.658920 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.659173 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.659745 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.660180 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:03:45.665086 28278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 16:03:45 dist-test-slave-b69v krb5kdc[28011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:45.669212 28278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:45.678822 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.656341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:37377 (local address 127.0.0.1:47196)
0929 16:03:45.656640 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:45.656664 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:45.656786 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:45.657338 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:45.657346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:45.657942 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:45.658185 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.658198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.659853 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.659862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.661695 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:45.661714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.662744 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.662752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:45.663036 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.663744 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:45.663774 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:45.667433 (+ 3659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:45.675155 (+ 7722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.675163 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.675170 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.675853 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.676426 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:45.676434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:45.676440 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:45.676661 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:45.677484 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:45.677492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:45.677689 (+ 197us) client_negotiation.cc:769] Sending connection context
0929 16:03:45.677856 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 16:03:45.678240 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 16:03:45.679633 28244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.612416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.5.117.60:37377)
0929 16:03:45.612671 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.612680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.636734 (+ 24054us) server_negotiation.cc:407] Connection header received
0929 16:03:45.637034 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.637040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.637165 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.637715 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.642194 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.643327 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.661935 (+ 18608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.663222 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.663336 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.666680 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.666686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.666691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.666729 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.672440 (+ 5711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.673857 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.673867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.673873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.674223 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.675243 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.675255 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.675261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.676241 (+ 980us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.676784 (+ 543us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.678097 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.678650 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":557,"server-negotiator.queue_time_us":87}
I20260929 16:03:45.680208 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:45.656425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.5.117.60:37377)
0929 16:03:45.656627 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:45.656633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:45.656903 (+ 270us) server_negotiation.cc:407] Connection header received
0929 16:03:45.656959 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:45.656963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:45.657039 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:45.657139 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:45.658365 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.659663 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.661865 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:45.662600 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:45.662726 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:45.667723 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:45.667732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:45.667740 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:45.667789 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:45.674967 (+ 7178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.676083 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.676092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.676098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.676266 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:45.676837 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:45.676842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:45.676845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:45.677295 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:45.679269 (+ 1974us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:45.679420 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:03:45.679746 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":65}
I20260929 16:03:45.688210 28165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: Bootstrap starting.
I20260929 16:03:45.697973 28165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:45.699702 28165 log.cc:824] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:45.716557 28165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: No bootstrap required, opened a new log
W20260929 16:03:45.722545 28287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:45.722675 28287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 16:03:45.723239 28194 tablet.cc:2369] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:45.726361 28215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [term 1 FOLLOWER]: Refusing update from remote peer fd7b45a978df4c38b756394cba2dfa35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:45.727260 28273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:45.734664 28165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.735195 28165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e6c2dd2386d4c2dbce9c2caa6096239, State: Initialized, Role: FOLLOWER
I20260929 16:03:45.735788 28165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } }
I20260929 16:03:45.738345 28165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:45.738431 28140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd7b45a978df4c38b756394cba2dfa35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239"
I20260929 16:03:45.738870 28140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:45.743139 28140 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd7b45a978df4c38b756394cba2dfa35 in term 1.
I20260929 16:03:45.744131 28139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [term 1 FOLLOWER]: Refusing update from remote peer fd7b45a978df4c38b756394cba2dfa35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:45.746403 28273 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:45.737489 28289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.752189 28289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.721167 28173 master_runner.cc:186] Error getting master registration for 127.5.117.61:44215: OK, instance_id { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" instance_seqno: 1790697824924360 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:45.752537 28173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:45.766935 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7b45a978df4c38b756394cba2dfa35. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.767297 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.771448 28279 mvcc.cc:205] Tried to move back new op lower bound from 7334698294113038336 to 7334698293877452800. Current Snapshot: MvccSnapshot[applied={T|T < 7334698294113038336}]
I20260929 16:03:45.780809 28289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7b45a978df4c38b756394cba2dfa35. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.781172 28289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.781795 28271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.782058 28271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.787734 28250 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.787981 28278 catalog_manager.cc:1347] Generated new cluster ID: f5eb5e4623b944238d64994d810fe7ce
I20260929 16:03:45.788138 28250 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:45.788144 28278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:45.798586 28273 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.798926 28273 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:45.801450 28294 mvcc.cc:205] Tried to move back new op lower bound from 7334698294113038336 to 7334698293877452800. Current Snapshot: MvccSnapshot[applied={T|T < 7334698294113038336}]
I20260929 16:03:45.810879 28289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd7b45a978df4c38b756394cba2dfa35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7b45a978df4c38b756394cba2dfa35" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 46867 } } peers { permanent_uuid: "8e6c2dd2386d4c2dbce9c2caa6096239" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 44215 } } peers { permanent_uuid: "498d01c81d4449aeabfc32700b854835" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 37377 } } }
I20260929 16:03:45.811195 28289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:45.825982 28278 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:45.829000 28278 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:45.862711 28278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd7b45a978df4c38b756394cba2dfa35: Generated new TSK 0
I20260929 16:03:45.864575 28278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:45.903689 28309 catalog_manager.cc:1259] Loaded cluster ID: f5eb5e4623b944238d64994d810fe7ce
I20260929 16:03:45.903837 28309 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: loading cluster ID for follower catalog manager: success
I20260929 16:03:45.908475 28309 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: acquiring CA information for follower catalog manager: success
I20260929 16:03:45.911357 28309 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e6c2dd2386d4c2dbce9c2caa6096239: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:03:46.038085 28247 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.004024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.5.117.62:46867)
0929 16:03:46.004331 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.004342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.034164 (+ 29822us) server_negotiation.cc:407] Connection header received
0929 16:03:46.036172 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.036179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.036281 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.036369 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.037738 (+ 1369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41774: BlockingRecv error: recv got EOF from 127.0.0.1:41774 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
W20260929 16:03:46.041162 28249 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.006359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.5.117.61:44215)
0929 16:03:46.006576 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.006582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.039155 (+ 32573us) server_negotiation.cc:407] Connection header received
0929 16:03:46.039560 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.039565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.039666 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.039739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.040879 (+ 1140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54476: BlockingRecv error: recv got EOF from 127.0.0.1:54476 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 16:03:46.043654 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.016279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.5.117.60:37377)
0929 16:03:46.016507 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.016514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.041945 (+ 25431us) server_negotiation.cc:407] Connection header received
0929 16:03:46.042184 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.042191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.042288 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.042368 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.043368 (+ 1000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47198: BlockingRecv error: recv got EOF from 127.0.0.1:47198 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:03:46.125105 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.106742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.5.117.61:44215)
0929 16:03:46.106957 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.106964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.116707 (+ 9743us) server_negotiation.cc:407] Connection header received
0929 16:03:46.117020 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.117028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.117271 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.117462 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.118235 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.119458 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.121624 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.122489 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.122615 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.123387 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.124197 (+ 810us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.124320 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.124580 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.124684 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:03:46.170478 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.155445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.5.117.60:37377)
0929 16:03:46.155645 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.155651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.168002 (+ 12351us) server_negotiation.cc:407] Connection header received
0929 16:03:46.168631 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.168641 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.168794 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.168886 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.170206 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47214: BlockingRecv error: recv got EOF from 127.0.0.1:47214 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:46.221159 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.203366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.5.117.61:44215)
0929 16:03:46.203592 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.203601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.214251 (+ 10650us) server_negotiation.cc:407] Connection header received
0929 16:03:46.214835 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.214846 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.214969 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.215056 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.215771 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.216891 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.218778 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.219418 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.219528 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.220192 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.220370 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.220440 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.220693 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.220789 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:03:46.273451 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.259397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.5.117.60:37377)
0929 16:03:46.259641 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.259650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.270877 (+ 11227us) server_negotiation.cc:407] Connection header received
0929 16:03:46.271494 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.271501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.271625 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.271717 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.273141 (+ 1424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47218: BlockingRecv error: recv got EOF from 127.0.0.1:47218 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:03:46.314100 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.305265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.5.117.61:44215)
0929 16:03:46.305523 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.305530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.305559 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:03:46.305875 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.305889 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.306009 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.306086 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.306934 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.308385 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.311340 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.312173 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.312417 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.313050 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.313255 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.313362 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.313576 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.313689 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 16:03:46.327455 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.324173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.5.117.60:37377)
0929 16:03:46.324461 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.324468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.325128 (+ 660us) server_negotiation.cc:407] Connection header received
0929 16:03:46.325540 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.325547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.325667 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.325767 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.327144 (+ 1377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47232: BlockingRecv error: recv got EOF from 127.0.0.1:47232 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:03:46.361578 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.337817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54516 (local address 127.5.117.61:44215)
0929 16:03:46.338127 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.338135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.353108 (+ 14973us) server_negotiation.cc:407] Connection header received
0929 16:03:46.353580 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.353589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.353710 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.353803 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.354799 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.356151 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.358861 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.359617 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.359738 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.360491 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.360713 (+ 222us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.360791 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.361079 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.361199 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
W20260929 16:03:46.381095 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.378589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.5.117.60:37377)
0929 16:03:46.378837 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.378844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.379371 (+ 527us) server_negotiation.cc:407] Connection header received
0929 16:03:46.379579 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.379585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.379700 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.379836 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.380777 (+ 941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47248: BlockingRecv error: recv got EOF from 127.0.0.1:47248 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 16:03:46.399998 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.392451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.5.117.61:44215)
0929 16:03:46.392642 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.392648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.393176 (+ 528us) server_negotiation.cc:407] Connection header received
0929 16:03:46.393341 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.393347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.393489 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.393600 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.394100 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.395364 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.397535 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.398302 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.398449 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.399047 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.399209 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.399320 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.399504 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.399604 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:03:46.409902 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.407324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.5.117.60:37377)
0929 16:03:46.407523 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.407529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.408346 (+ 817us) server_negotiation.cc:407] Connection header received
0929 16:03:46.408493 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.408499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.408599 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.408717 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.409640 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47264: BlockingRecv error: recv got EOF from 127.0.0.1:47264 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:03:46.435611 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.429145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.5.117.61:44215)
0929 16:03:46.429366 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.429372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.429677 (+ 305us) server_negotiation.cc:407] Connection header received
0929 16:03:46.429854 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.429860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.429973 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.430100 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.430519 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.431786 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.433506 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.434256 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.434409 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.434759 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.434902 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.435023 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.435148 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.435239 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:03:46.444319 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.441917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.5.117.60:37377)
0929 16:03:46.442184 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.442190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.442641 (+ 451us) server_negotiation.cc:407] Connection header received
0929 16:03:46.442852 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.442857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.442935 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.443019 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.444017 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47266: BlockingRecv error: recv got EOF from 127.0.0.1:47266 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:03:46.487538 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.481040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.5.117.61:44215)
0929 16:03:46.481318 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.481326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.481668 (+ 342us) server_negotiation.cc:407] Connection header received
0929 16:03:46.481933 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.481941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.482039 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.482133 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.482642 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.483521 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.485607 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.486077 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.486175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.486664 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.486823 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.486915 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.487097 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.487168 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 16:03:46.497215 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.494773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.5.117.60:37377)
0929 16:03:46.495004 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.495011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.495686 (+ 675us) server_negotiation.cc:407] Connection header received
0929 16:03:46.495775 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.495781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.495889 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.496004 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.496974 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47276: BlockingRecv error: recv got EOF from 127.0.0.1:47276 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:03:46.570398 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.564951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.5.117.61:44215)
0929 16:03:46.565228 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.565236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.565476 (+ 240us) server_negotiation.cc:407] Connection header received
0929 16:03:46.565733 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.565738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.565818 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.565930 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.566371 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.567175 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.568579 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.569046 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.569191 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.569614 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.569725 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.569829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.570003 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.570089 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 16:03:46.578259 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.576241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47288 (local address 127.5.117.60:37377)
0929 16:03:46.576500 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.576508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.576753 (+ 245us) server_negotiation.cc:407] Connection header received
0929 16:03:46.576970 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.576976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.577067 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.577179 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.577996 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47288: BlockingRecv error: recv got EOF from 127.0.0.1:47288 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:03:46.716746 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.710624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.5.117.61:44215)
0929 16:03:46.710992 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.711003 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.711377 (+ 374us) server_negotiation.cc:407] Connection header received
0929 16:03:46.711677 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.711683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.711766 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.711883 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.712570 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.713384 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.714785 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.715241 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.715387 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.715871 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.715998 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.716173 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.716338 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.716427 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260929 16:03:46.725342 28241 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:46.723242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.5.117.60:37377)
0929 16:03:46.723482 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.723487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.723635 (+ 148us) server_negotiation.cc:407] Connection header received
0929 16:03:46.723841 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.723844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.723945 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.724095 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:46.725071 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47294: BlockingRecv error: recv got EOF from 127.0.0.1:47294 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:03:46.725967 28287 catalog_manager.cc:1259] Loaded cluster ID: f5eb5e4623b944238d64994d810fe7ce
I20260929 16:03:46.726053 28287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: loading cluster ID for follower catalog manager: success
I20260929 16:03:46.729187 28287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: acquiring CA information for follower catalog manager: success
I20260929 16:03:46.731216 28287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 498d01c81d4449aeabfc32700b854835: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:46.991343 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.985747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.5.117.61:44215)
0929 16:03:46.986042 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.986049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.986256 (+ 207us) server_negotiation.cc:407] Connection header received
0929 16:03:46.986554 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.986559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.986658 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.986774 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.987263 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.988066 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.989467 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.989879 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:46.990032 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:46.990414 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:46.990610 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:46.990714 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:46.990875 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:03:46.990972 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 16:03:47.003736 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:46.997973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.5.117.60:37377)
0929 16:03:46.998226 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:46.998233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:46.998255 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:03:46.998374 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:46.998380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:46.998607 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:46.998750 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:46.999173 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:46.999940 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.001304 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.001738 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.001874 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:47.002402 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:47.003084 (+ 682us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:47.003267 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:47.003325 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:03:47.003416 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:03:47.028074 28241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:47.011425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47306 (local address 127.5.117.60:37377)
0929 16:03:47.011613 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:47.011619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:47.021627 (+ 10008us) server_negotiation.cc:407] Connection header received
0929 16:03:47.021841 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:47.021850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:47.021980 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:47.022089 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:47.023022 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.024273 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.025906 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.026563 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.026700 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:47.027241 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:47.027392 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:47.027489 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:47.027660 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 16:03:47.027748 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:03:47.029197 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:47.011027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.5.117.62:46867)
0929 16:03:47.020744 (+ 9717us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:47.020751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:47.021627 (+ 876us) server_negotiation.cc:407] Connection header received
0929 16:03:47.021825 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:47.021834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:47.022165 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:47.022394 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:47.023044 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.024273 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.026155 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.026812 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.026962 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:47.027694 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:47.028270 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:47.028382 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:47.028619 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:03:47.028716 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9528,"thread_start_us":9362,"threads_started":1}
I20260929 16:03:47.029722 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:47.011473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.5.117.61:44215)
0929 16:03:47.011661 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:47.011670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:47.022416 (+ 10746us) server_negotiation.cc:407] Connection header received
0929 16:03:47.022729 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:47.022738 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:47.022852 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:47.022922 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:47.023590 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.024496 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.027404 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:47.028108 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:47.028270 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:47.028938 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:47.029050 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:47.029163 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:47.029306 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 16:03:47.029396 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:47.036993 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28024
I20260929 16:03:47.038281 28086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:47.164609 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28024
I20260929 16:03:47.194211 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28098
I20260929 16:03:47.195564 28160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:47.317380 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28098
I20260929 16:03:47.344349 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28173
I20260929 16:03:47.345914 28235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:47.467578 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28173
2026-09-29T16:03:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5130 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:47 dist-test-slave-b69v krb5kdc[28434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:47 dist-test-slave-b69v krb5kdc[28434](info): set up 2 sockets
Sep 29 16:03:47 dist-test-slave-b69v krb5kdc[28434](info): commencing operation
krb5kdc: starting...
W20260929 16:03:48.594686 5588 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/test-user.keytab.
Sep 29 16:03:48 dist-test-slave-b69v krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.117.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
Entry for principal kudu/127.5.117.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab.
I20260929 16:03:48.947978 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:44279
--webserver_interface=127.5.117.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:49.320427 28447 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:49.320859 28447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:49.320997 28447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:49.321149 28447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:49.328593 28447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:49.328680 28447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:49.328815 28447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:49.328903 28447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:49.329003 28447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:49.339555 28447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.62.keytab
--principal=kudu/127.5.117.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.62:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.117.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:49.341251 28447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:49.342877 28447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:49.356680 28455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:49.356616 28453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:49.356925 28452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:49.357478 28447 server_base.cc:1034] running on GCE node
I20260929 16:03:49.358433 28447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:49.360196 28447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:49.361419 28447 hybrid_clock.cc:648] HybridClock initialized: now 1790697829361355 us; error 46 us; skew 500 ppm
I20260929 16:03:49.361724 28447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:49 dist-test-slave-b69v krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:49.365216 28447 init.cc:377] Logged in from keytab as kudu/127.5.117.62@KRBTEST.COM (short username kudu)
I20260929 16:03:49.366817 28447 webserver.cc:460] Webserver started at http://127.5.117.62:46881/ using document root <none> and password file <none>
I20260929 16:03:49.367681 28447 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:49.367800 28447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:49.368242 28447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:49.372161 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data/instance:
uuid: "b63c7d6c07424876b7dad32b5a40f7d7"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "294ec4c2c2e1ef7ac016c9e948e063af"
server_key_iv: "5860b0aea6f44c6f4cc7cb9d8f0d7b94"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.373111 28447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance:
uuid: "b63c7d6c07424876b7dad32b5a40f7d7"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "294ec4c2c2e1ef7ac016c9e948e063af"
server_key_iv: "5860b0aea6f44c6f4cc7cb9d8f0d7b94"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.379041 28447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:03:49.382938 28462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:49.384457 28447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:49.384649 28447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal
uuid: "b63c7d6c07424876b7dad32b5a40f7d7"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "294ec4c2c2e1ef7ac016c9e948e063af"
server_key_iv: "5860b0aea6f44c6f4cc7cb9d8f0d7b94"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.384917 28447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:49.410193 28447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:49.416435 28447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:49.416815 28447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:49.441555 28447 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.62:44279
I20260929 16:03:49.441586 28513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.62:44279 every 8 connection(s)
I20260929 16:03:49.443421 28447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:49.448730 28514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:49.451915 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28447
I20260929 16:03:49.452261 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-0/wal/instance
I20260929 16:03:49.452836 5588 external_mini_cluster.cc:1428] Setting key 0364eee8e8cbc550ea3ce3c362ca4985
I20260929 16:03:49.457798 28514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:49 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:49.489629 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.462027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:44279 (local address 127.0.0.1:58044)
0929 16:03:49.463131 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.463200 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:49.465888 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:49.467313 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:49.467319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:49.469009 (+ 1690us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:49.469694 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:49.469733 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:49.471704 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:49.471714 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:49.474054 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:49.474079 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:49.475425 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:49.475435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:49.475703 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:49.476711 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:49.476751 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:49.479922 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:49.484776 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:49.484787 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:49.484798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:49.485245 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:49.485761 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:49.485764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:49.485767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:49.485938 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:49.487067 (+ 1129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:49.487081 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:49.487342 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 16:03:49.487586 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 16:03:49.488152 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":471,"threads_started":1}
I20260929 16:03:49.489671 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.463726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.5.117.62:44279)
0929 16:03:49.464737 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:49.464751 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:49.464826 (+ 75us) server_negotiation.cc:407] Connection header received
0929 16:03:49.466462 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:49.466474 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:49.466965 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:49.467263 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:49.470005 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:49.471488 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:49.474377 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:49.475100 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:49.475290 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:49.480253 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:49.480267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:49.480292 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:49.480337 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:49.484638 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:49.485410 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:49.485422 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:49.485428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:49.485638 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:49.486066 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:49.486074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:49.486080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:49.486913 (+ 833us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:49.487144 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:49.487787 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 16:03:49.488373 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":211,"server-negotiator.queue_time_us":719,"thread_start_us":528,"threads_started":1}
WARNING: no policy specified for kudu/127.5.117.61@KRBTEST.COM; defaulting to no policy
I20260929 16:03:49.493405 28514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:49.495010 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.494338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60570)
0929 16:03:49.494594 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.494783 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:03:49.495375 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.61@KRBTEST.COM" created.
I20260929 16:03:49.497462 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.497044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60576)
0929 16:03:49.497256 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.497332 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:03:49.497660 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:35383: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
Entry for principal kudu/127.5.117.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab.
I20260929 16:03:49.535761 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:35383
--webserver_interface=127.5.117.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:49.546979 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } attempt: 1
I20260929 16:03:49.548307 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.547696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60582)
0929 16:03:49.547953 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.548094 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:03:49.549423 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.548971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60592)
0929 16:03:49.549171 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.549259 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:03:49.549654 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:35383: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
I20260929 16:03:49.617936 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } attempt: 2
I20260929 16:03:49.619611 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.618909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60606)
0929 16:03:49.619203 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.619306 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:03:49.620882 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.620421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60614)
0929 16:03:49.620581 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.620658 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:03:49.621176 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:35383: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
I20260929 16:03:49.712507 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } attempt: 3
I20260929 16:03:49.714030 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.713534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60630)
0929 16:03:49.713757 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.713866 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:03:49.715051 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.714704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60632)
0929 16:03:49.714856 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.714924 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:03:49.715238 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:35383: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
I20260929 16:03:49.858578 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } attempt: 4
I20260929 16:03:49.859936 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.859368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60646)
0929 16:03:49.859668 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.859777 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 16:03:49.860931 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:49.860597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60656)
0929 16:03:49.860735 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:49.860817 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:03:49.861101 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.61:35383: Network error: Client connection negotiation failed: client connection to 127.5.117.61:35383: connect: Connection refused (error 111)
W20260929 16:03:49.915637 28521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:49.916103 28521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:49.916241 28521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:49.916394 28521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:49.923863 28521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:49.924170 28521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:49.924495 28521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:49.924666 28521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:49.924784 28521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:49.935773 28521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.61.keytab
--principal=kudu/127.5.117.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.61:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.117.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:49.937294 28521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:49.938812 28521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:49.952239 28526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:49.952322 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:49.952363 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:49.953655 28521 server_base.cc:1034] running on GCE node
I20260929 16:03:49.954456 28521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:49.955968 28521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:49.957253 28521 hybrid_clock.cc:648] HybridClock initialized: now 1790697829957192 us; error 32 us; skew 500 ppm
I20260929 16:03:49.957619 28521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:49 dist-test-slave-b69v krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:49.961161 28521 init.cc:377] Logged in from keytab as kudu/127.5.117.61@KRBTEST.COM (short username kudu)
I20260929 16:03:49.962773 28521 webserver.cc:460] Webserver started at http://127.5.117.61:42013/ using document root <none> and password file <none>
I20260929 16:03:49.963533 28521 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:49.963673 28521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:49.964030 28521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:49.967353 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data/instance:
uuid: "156a4acb21c249faa1773a7cce3a8df9"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "500c6f5d7ac79b78a9af4f6693687998"
server_key_iv: "9394e12bcb6412a41356ccc0aec45a7d"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.968287 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal/instance:
uuid: "156a4acb21c249faa1773a7cce3a8df9"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "500c6f5d7ac79b78a9af4f6693687998"
server_key_iv: "9394e12bcb6412a41356ccc0aec45a7d"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.973718 28521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:03:49.977275 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:49.978619 28521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:49.978808 28521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal
uuid: "156a4acb21c249faa1773a7cce3a8df9"
format_stamp: "Formatted at 2026-09-29 16:03:49 on dist-test-slave-b69v"
server_key: "500c6f5d7ac79b78a9af4f6693687998"
server_key_iv: "9394e12bcb6412a41356ccc0aec45a7d"
server_key_version: "encryptionkey@0"
I20260929 16:03:49.979050 28521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:49.995689 28521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:49.999619 28521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:49.999902 28521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:50.019256 28521 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.61:35383
I20260929 16:03:50.019265 28587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.61:35383 every 8 connection(s)
I20260929 16:03:50.020720 28521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/data/info.pb
I20260929 16:03:50.024621 28588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:50.027194 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28521
I20260929 16:03:50.027412 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-1/wal/instance
I20260929 16:03:50.028019 5588 external_mini_cluster.cc:1428] Setting key 7a26457750edb1528385654cb94253b2
I20260929 16:03:50.034150 28588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:50.064339 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.038583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:44279 (local address 127.0.0.1:58050)
0929 16:03:50.039544 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.039625 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.042821 (+ 3196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.043874 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.043888 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.044853 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.045789 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.045821 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.047761 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.047766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.049130 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.049143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.050372 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.050377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.050576 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.051507 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.051532 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.054902 (+ 3370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.059287 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.059295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.059310 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.059981 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.060604 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.060612 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.060616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.060883 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.061713 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.061721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.062138 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.062419 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.062904 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":362,"threads_started":1}
I20260929 16:03:50.065443 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.038002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.5.117.62:44279)
0929 16:03:50.038617 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.038625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.039805 (+ 1180us) server_negotiation.cc:407] Connection header received
0929 16:03:50.043291 (+ 3486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.043297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.043421 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.043571 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.046108 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.047563 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.049287 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.050135 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.050304 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.055238 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.055246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.055254 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.055324 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.059109 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.060170 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.060179 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.060184 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.060335 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.061041 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.061048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.061053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.061458 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.061633 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.064858 (+ 3225us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.065086 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":309,"threads_started":1}
I20260929 16:03:50.067898 28588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.117.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.117.60@KRBTEST.COM" created.
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:50.093981 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.069171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.5.117.61:35383)
0929 16:03:50.069851 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.069859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.069898 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:03:50.070110 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.070123 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.070389 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.070582 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.071462 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.073620 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.075612 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.076760 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.076909 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.086640 (+ 9731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.086653 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.086670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.086701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.090135 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.091032 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.091209 (+ 177us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.091212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.091371 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.091843 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.091847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.091849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.092585 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.092762 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.093269 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.093647 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":495,"thread_start_us":346,"threads_started":1}
I20260929 16:03:50.094028 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.068778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60670)
0929 16:03:50.069800 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.069825 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.069939 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.070620 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.070625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.071094 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.071262 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.071276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.073817 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.073824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.075451 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.075468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.076910 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.076916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.077206 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.083701 (+ 6495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.083720 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.086341 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.090287 (+ 3946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.090295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.090302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.090870 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.091526 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.091534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.091540 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.091714 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.092770 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.092778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.092963 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.093132 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.093518 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":25}
I20260929 16:03:50.097468 28588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:50.099460 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.098721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56462)
0929 16:03:50.099032 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.099207 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260929 16:03:50.099963 28538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.102097 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.101575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56470)
0929 16:03:50.101784 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.101903 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 16:03:50.102437 28588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
Entry for principal kudu/127.5.117.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab.
I20260929 16:03:50.116197 5588 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
/tmp/dist-test-task0NOJVQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:46193
--webserver_interface=127.5.117.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.117.20:34633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:50.151944 28588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 1
I20260929 16:03:50.153507 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.152870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56476)
0929 16:03:50.153148 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.153289 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 16:03:50.154687 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.154290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56490)
0929 16:03:50.154468 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.154564 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:03:50.154948 28588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.160357 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } attempt: 5
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:50.176368 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.161191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60674)
0929 16:03:50.161432 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.161452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.161585 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.161999 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.162003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.162449 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.162664 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.162673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.164206 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.164213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.165164 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.165176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.166467 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.166472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.166645 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.167349 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.167367 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.170184 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.173998 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.174002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.174005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.174387 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.174827 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.174830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.174833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.174962 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.175602 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.175605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.175715 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.175828 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.176043 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:03:50.176597 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.161305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.5.117.61:35383)
0929 16:03:50.161505 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.161512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.161538 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:03:50.161745 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.161750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.161843 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.161968 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.162818 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.163945 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.165398 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.166218 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.166471 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.170387 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.170394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.170398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.170439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.173858 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.174515 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.174521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.174524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.174693 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.175068 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.175073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.175078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.175478 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.175650 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.176000 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.176286 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:50.177639 28514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:50.178663 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.178313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56496)
0929 16:03:50.178466 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.178548 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:03:50.179579 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.179260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56510)
0929 16:03:50.179414 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.179472 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:03:50.179804 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.223213 28588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 2
I20260929 16:03:50.224709 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.224156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56520)
0929 16:03:50.224377 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.224495 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:03:50.225732 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.225384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56524)
0929 16:03:50.225527 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.225605 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:50.225929 28588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.231031 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 1
I20260929 16:03:50.232005 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.231607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56538)
0929 16:03:50.231775 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.231857 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:03:50.232976 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.232655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56540)
0929 16:03:50.232812 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.232877 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:03:50.233206 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.301458 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 2
I20260929 16:03:50.302789 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.302243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56550)
0929 16:03:50.302487 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.302588 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:03:50.303807 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.303451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56560)
0929 16:03:50.303634 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.303701 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:50.304139 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.317169 28588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 3
I20260929 16:03:50.318421 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.317872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56570)
0929 16:03:50.318090 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.318207 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:03:50.319576 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.319104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56582)
0929 16:03:50.319344 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.319441 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260929 16:03:50.319840 28588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.410375 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 3
I20260929 16:03:50.411701 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.411229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56590)
0929 16:03:50.411439 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.411531 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:03:50.412753 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.412352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56594)
0929 16:03:50.412589 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.412660 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 16:03:50.412983 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.463172 28588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 4
I20260929 16:03:50.464907 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.464179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56610)
0929 16:03:50.464504 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.464627 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 16:03:50.466528 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.465886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56624)
0929 16:03:50.466127 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.466314 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 16:03:50.466867 28588 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
W20260929 16:03:50.491894 28596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:50.492401 28596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:50.492514 28596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:50.492635 28596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:50.500699 28596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:50.500824 28596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:50.500885 28596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:50.500964 28596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:50.501055 28596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:50.511615 28596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.117.20:34633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.117.62:44279,127.5.117.61:35383,127.5.117.60:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/krb5kdc/kudu_127.5.117.60.keytab
--principal=kudu/127.5.117.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.117.60:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.117.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 14:53:52 UTC on ccycloud.quasar-tnttyx.root.comops.site
build id 4680448
ASAN enabled
I20260929 16:03:50.513547 28596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:50.515847 28596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:50.531656 28602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:50.531656 28604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:50.531697 28601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:50.533092 28596 server_base.cc:1034] running on GCE node
I20260929 16:03:50.534163 28596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:50.536299 28596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:50.537575 28596 hybrid_clock.cc:648] HybridClock initialized: now 1790697830537509 us; error 54 us; skew 500 ppm
I20260929 16:03:50.538005 28596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:50.541772 28596 init.cc:377] Logged in from keytab as kudu/127.5.117.60@KRBTEST.COM (short username kudu)
I20260929 16:03:50.543500 28596 webserver.cc:460] Webserver started at http://127.5.117.60:33919/ using document root <none> and password file <none>
I20260929 16:03:50.544374 28596 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:50.544469 28596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:50.544778 28596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:50.548012 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data/instance:
uuid: "4e1cc2abea8643d0bcec2581adb5dc97"
format_stamp: "Formatted at 2026-09-29 16:03:50 on dist-test-slave-b69v"
server_key: "72d829ae06df3dd8cc1313c2dd28031d"
server_key_iv: "5f0b2ee2665ac419b924f2279e0fc46d"
server_key_version: "encryptionkey@0"
I20260929 16:03:50.548877 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal/instance:
uuid: "4e1cc2abea8643d0bcec2581adb5dc97"
format_stamp: "Formatted at 2026-09-29 16:03:50 on dist-test-slave-b69v"
server_key: "72d829ae06df3dd8cc1313c2dd28031d"
server_key_iv: "5f0b2ee2665ac419b924f2279e0fc46d"
server_key_version: "encryptionkey@0"
I20260929 16:03:50.554390 28596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:50.558359 28611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:50.560148 28596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:03:50.560331 28596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data,/tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal
uuid: "4e1cc2abea8643d0bcec2581adb5dc97"
format_stamp: "Formatted at 2026-09-29 16:03:50 on dist-test-slave-b69v"
server_key: "72d829ae06df3dd8cc1313c2dd28031d"
server_key_iv: "5f0b2ee2665ac419b924f2279e0fc46d"
server_key_version: "encryptionkey@0"
I20260929 16:03:50.560549 28596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:50.590224 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 4
I20260929 16:03:50.590688 28596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:50.591562 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.591021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56638)
0929 16:03:50.591228 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.591332 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:03:50.592672 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.592347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56648)
0929 16:03:50.592484 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.592554 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:03:50.592900 28514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.117.60:46193: Network error: Client connection negotiation failed: client connection to 127.5.117.60:46193: connect: Connection refused (error 111)
I20260929 16:03:50.596040 28596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:50.596400 28596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:50.619172 28596 rpc_server.cc:307] RPC server started. Bound to: 127.5.117.60:46193
I20260929 16:03:50.619149 28662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.117.60:46193 every 8 connection(s)
I20260929 16:03:50.620777 28596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/data/info.pb
I20260929 16:03:50.624835 28663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:50.630836 5588 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu as pid 28596
I20260929 16:03:50.631080 5588 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0NOJVQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697441705944-5588-0/minicluster-data/master-2/wal/instance
I20260929 16:03:50.631538 5588 external_mini_cluster.cc:1428] Setting key 58f203842cf517f2e63939e8f7022937
I20260929 16:03:50.631826 28663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:50.651681 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.634918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.5.117.62:44279)
0929 16:03:50.636432 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.636438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.636457 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:03:50.637100 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.637104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.637182 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.637306 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.638662 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.639469 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.641050 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.641460 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.641566 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.645128 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.645136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.645141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.645188 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.648575 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.649314 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.649319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.649321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.649464 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.649954 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.649958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.649961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.650280 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.650417 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.651022 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.651250 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388,"thread_start_us":1261,"threads_started":1}
I20260929 16:03:50.652590 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.633587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.5.117.60:46193)
0929 16:03:50.634391 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.634401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.634439 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:03:50.636828 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.636854 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.637156 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.637348 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.638689 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.639945 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.641080 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.641600 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.641753 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.646041 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.646054 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.646068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.646109 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.649217 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.649954 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.649961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.649964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.650084 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.650416 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.650421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.650424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.650754 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.651033 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.651289 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.651497 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":403,"threads_started":1}
I20260929 16:03:50.652623 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.634805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.62:44279 (local address 127.0.0.1:58066)
0929 16:03:50.635656 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.635704 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.636828 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.637359 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.637365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.638126 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.638490 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.638506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.639622 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.639627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.640902 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.640914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.641677 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.641685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.641820 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.642525 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.642549 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.644929 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.648684 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.648693 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.648699 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.649199 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.649624 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.649628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.649632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.649848 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.650406 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.650413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.650683 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.650919 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.651360 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":402,"threads_started":1}
I20260929 16:03:50.654287 28663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.62@KRBTEST.COM
I20260929 16:03:50.667424 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.654847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.61:35383 (local address 127.0.0.1:60678)
0929 16:03:50.655032 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.655045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.655126 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.655745 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.655750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.656212 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.656399 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.656409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.657345 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.657349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.658338 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.658351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.658978 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.658985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.659146 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.659600 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.659614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.661760 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.664947 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.664953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.664959 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.665403 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.665797 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.665800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.665802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.665935 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.666549 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.666552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.666698 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.666858 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.667123 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 16:03:50.667429 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.655201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.5.117.61:35383)
0929 16:03:50.655404 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.655410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.655430 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:03:50.655498 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.655503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.655600 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.655710 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.656534 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.657219 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.658470 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.658853 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.658935 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.661934 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.661938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.661941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.661971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.664852 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.665513 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.665520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.665525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.665695 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.666037 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.666042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.666044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.666444 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.666610 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.666945 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.667101 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:03:50.668409 28663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:50.669198 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.655497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.5.117.62:44279)
0929 16:03:50.655651 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.655657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.655674 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:03:50.655739 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.655742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.655812 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.655896 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.657131 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.657764 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.659067 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.659575 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.659656 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.663593 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.663601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.663607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.663646 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.666636 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.667324 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.667329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.667332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.667469 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.667944 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.667948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.667952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.668317 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.668529 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.668822 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.668966 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697830, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.60@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.117.61@KRBTEST.COM
I20260929 16:03:50.680989 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.669082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56676)
0929 16:03:50.669380 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.669396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.669477 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.669787 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.669790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.670162 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.670318 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.670328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.671249 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.671256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.671970 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.671978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.672648 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.672653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.672917 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.673419 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.673438 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.675191 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.678562 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.678568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.678571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.678912 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.679276 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.679279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.679281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.679536 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.680075 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.680079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.680198 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.680347 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.680637 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":22}
I20260929 16:03:50.681056 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.669396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.5.117.60:46193)
0929 16:03:50.669512 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.669517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.669533 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:03:50.669594 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.669599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.669664 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.669762 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.670458 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.671142 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.672131 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.672516 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.672642 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.675361 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.675366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.675370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.675404 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.678442 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.679021 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.679027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.679030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.679162 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.679675 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.679682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.679690 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.679943 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.680101 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.680415 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.680755 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":199,"server-negotiator.queue_time_us":40}
I20260929 16:03:50.685079 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.670711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.5.117.61:35383)
0929 16:03:50.670848 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.670852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.670867 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:03:50.670962 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.670966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.671028 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.671117 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.672161 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.672859 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.674224 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.674806 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.674914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.679662 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.679667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.679672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.679709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.682521 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.683313 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.683317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.683319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.683504 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.683882 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.683885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.683887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.684219 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.684419 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.684690 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.684835 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:03:50.687121 28663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: Bootstrap starting.
I20260929 16:03:50.689479 5588 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:50.690922 28663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:50.692219 28663 log.cc:824] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:50.695138 28663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: No bootstrap required, opened a new log
I20260929 16:03:50.691954 28489 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b63c7d6c07424876b7dad32b5a40f7d7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58076
I20260929 16:03:50.702121 28663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.702473 28663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1cc2abea8643d0bcec2581adb5dc97, State: Initialized, Role: FOLLOWER
I20260929 16:03:50.702984 28663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.704039 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:50.704483 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:50.705081 28663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:03:50.715226 28681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:50.715358 28681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:50.712989 28596 master_runner.cc:186] Error getting master registration for 127.5.117.62:44279: OK, instance_id { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" instance_seqno: 1790697829434242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:50.724098 28596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:50.766124 28588 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 5
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.61@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:50.783077 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.766990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56682)
0929 16:03:50.767198 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.767222 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.767371 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.767755 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.767760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.768316 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.768498 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.768507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.769947 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.769952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.771248 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.771261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.772268 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.772275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.772522 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.773311 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.773336 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.776317 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.780261 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.780266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.780270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.780770 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.781139 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.781144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.781148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.781320 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.781995 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.782000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.782150 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.782310 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.782609 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:03:50.783094 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.767073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.5.117.60:46193)
0929 16:03:50.767412 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.767421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.767449 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:03:50.767532 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.767536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.767605 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.767723 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.768686 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.769784 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.771409 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.772132 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.772269 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.776534 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.776539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.776542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.776593 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.780122 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.780880 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.780888 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.780892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.781026 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.781426 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.781431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.781436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.781868 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.782036 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.782433 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.782682 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 16:03:50.788175 28588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: Bootstrap starting.
I20260929 16:03:50.791970 28588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:50.793428 28588 log.cc:824] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:50.796664 28588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: No bootstrap required, opened a new log
I20260929 16:03:50.803987 28588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.804409 28588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 156a4acb21c249faa1773a7cce3a8df9, State: Initialized, Role: FOLLOWER
I20260929 16:03:50.805004 28588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.806075 28684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:50.806509 28684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:50.807307 28588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:03:50.823622 28695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:50.823770 28695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:50.821593 28521 master_runner.cc:186] Error getting master registration for 127.5.117.62:44279: OK, instance_id { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" instance_seqno: 1790697829434242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:50.833640 28521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:03:50.849179 28617 tablet.cc:2369] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:50.870072 28514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } attempt: 5
Sep 29 16:03:50 dist-test-slave-b69v krb5kdc[28434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697829, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.117.62@KRBTEST.COM for kudu/127.5.117.60@KRBTEST.COM
I20260929 16:03:50.890846 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.871007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.117.60:46193 (local address 127.0.0.1:56696)
0929 16:03:50.871497 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:50.871521 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:50.871671 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:50.872613 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:50.872619 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:50.873265 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:50.873504 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.873517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.875191 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.875197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.876925 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:50.876942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.878056 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.878062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:50.878277 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.879098 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:50.879125 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:50.882547 (+ 3422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:50.887492 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.887500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.887508 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.888104 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.888631 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:50.888637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:50.888641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:50.888893 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:50.889657 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:50.889664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:50.889853 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 16:03:50.890023 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:03:50.890365 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314}
I20260929 16:03:50.890851 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.871941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.5.117.60:46193)
0929 16:03:50.872172 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.872179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:50.872202 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:03:50.872284 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:50.872290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:50.872410 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:50.872531 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:50.873727 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.875030 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.877079 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:50.877838 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:50.877999 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:50.882769 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:50.882777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:50.882782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:50.882834 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:50.887332 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.888231 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.888237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.888241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.888489 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:50.889022 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:50.889028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:50.889031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:50.889511 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:50.889687 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:50.890173 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 16:03:50.890440 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:03:50.897993 28514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7: Bootstrap starting.
I20260929 16:03:50.902940 28514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:50.904603 28514 log.cc:824] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:50.915758 28514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7: No bootstrap required, opened a new log
I20260929 16:03:50.924526 28514 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.924968 28514 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63c7d6c07424876b7dad32b5a40f7d7, State: Initialized, Role: FOLLOWER
I20260929 16:03:50.925642 28514 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.927147 28698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:50.927865 28698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:50.928534 28514 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:50.928642 28489 raft_consensus.cc:482] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:03:50.928855 28489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:50.932603 28489 raft_consensus.cc:504] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.935921 28563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "156a4acb21c249faa1773a7cce3a8df9"
I20260929 16:03:50.936537 28563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:50.940469 28563 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63c7d6c07424876b7dad32b5a40f7d7 in term 1.
I20260929 16:03:50.941246 28463 leader_election.cc:302] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 156a4acb21c249faa1773a7cce3a8df9, b63c7d6c07424876b7dad32b5a40f7d7; no voters:
I20260929 16:03:50.942938 28638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4e1cc2abea8643d0bcec2581adb5dc97"
I20260929 16:03:50.943485 28638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:50.947816 28638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b63c7d6c07424876b7dad32b5a40f7d7 in term 1.
I20260929 16:03:50.948633 28489 leader_election.cc:288] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [CANDIDATE]: Term 1 election: Requested vote from peers 156a4acb21c249faa1773a7cce3a8df9 (127.5.117.61:35383), 4e1cc2abea8643d0bcec2581adb5dc97 (127.5.117.60:46193)
I20260929 16:03:50.956090 28698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:50.966473 28698 raft_consensus.cc:686] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [term 1 LEADER]: Becoming Leader. State: Replica: b63c7d6c07424876b7dad32b5a40f7d7, State: Running, Role: LEADER
I20260929 16:03:50.967296 28698 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } }
I20260929 16:03:50.978520 28705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63c7d6c07424876b7dad32b5a40f7d7. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:50.979051 28705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:50.983793 28711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 16:03:51.002683 28542 tablet.cc:2369] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:51.022203 28711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:51.046360 28638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [term 1 FOLLOWER]: Refusing update from remote peer b63c7d6c07424876b7dad32b5a40f7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:51.047657 28705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:51.051928 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.990551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.5.117.62:44279)
0929 16:03:50.990758 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.990765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.010094 (+ 19329us) server_negotiation.cc:407] Connection header received
0929 16:03:51.010387 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.010394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.010511 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.010643 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.013277 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.014429 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.017251 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.018053 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.018179 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.043850 (+ 25671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:51.043860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:51.043867 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:51.043917 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:51.047949 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.049121 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.049128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.049131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.049312 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.049821 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.049826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.049829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.050265 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:51.050571 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.051271 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.051532 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:03:51.056475 28563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [term 1 FOLLOWER]: Refusing update from remote peer b63c7d6c07424876b7dad32b5a40f7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:51.057835 28705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:51.062415 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.025697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.5.117.61:35383)
0929 16:03:51.025905 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.025912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.036767 (+ 10855us) server_negotiation.cc:407] Connection header received
0929 16:03:51.037085 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.037092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.037201 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.037280 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.039369 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.040515 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.042711 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.043450 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.043567 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.053800 (+ 10233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:51.053810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:51.053818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:51.053869 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:51.058426 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.059408 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.059416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.059422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.059600 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.060178 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.060184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.060187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.060670 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:51.060970 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.061632 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.061909 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:03:51.063782 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:50.993405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.5.117.60:46193)
0929 16:03:50.993615 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:50.993622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.014176 (+ 20554us) server_negotiation.cc:407] Connection header received
0929 16:03:51.014459 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.014466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.014614 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.014713 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.030276 (+ 15563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.031519 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.039224 (+ 7705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.039974 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.040110 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.052520 (+ 12410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:51.052527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:51.052533 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:51.052576 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:51.056583 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.060210 (+ 3627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.060216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.060220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.060385 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:51.061132 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:51.061137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:51.061140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:51.061541 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:51.061770 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.063024 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.063294 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:51.066931 28719 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:03:51.081621 28684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63c7d6c07424876b7dad32b5a40f7d7. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.081967 28684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:51.082078 28721 mvcc.cc:205] Tried to move back new op lower bound from 7334698315946053632 to 7334698315661131776. Current Snapshot: MvccSnapshot[applied={T|T < 7334698315946053632}]
I20260929 16:03:51.099092 28698 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.099476 28698 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:51.105762 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63c7d6c07424876b7dad32b5a40f7d7. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.106163 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:51.110244 28684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.110591 28684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:51.115643 28670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.115967 28670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:51.116508 28705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63c7d6c07424876b7dad32b5a40f7d7" member_type: VOTER last_known_addr { host: "127.5.117.62" port: 44279 } } peers { permanent_uuid: "156a4acb21c249faa1773a7cce3a8df9" member_type: VOTER last_known_addr { host: "127.5.117.61" port: 35383 } } peers { permanent_uuid: "4e1cc2abea8643d0bcec2581adb5dc97" member_type: VOTER last_known_addr { host: "127.5.117.60" port: 46193 } } }
I20260929 16:03:51.116806 28705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:51.117871 28711 catalog_manager.cc:1347] Generated new cluster ID: 69997fa1a4e44605afd5f7208b179a65
I20260929 16:03:51.117998 28711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:51.150316 28711 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:51.152825 28711 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:51.174682 28711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b63c7d6c07424876b7dad32b5a40f7d7: Generated new TSK 0
I20260929 16:03:51.176038 28711 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:03:51.334607 28668 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.308484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.5.117.62:44279)
0929 16:03:51.308950 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.308959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.331991 (+ 23032us) server_negotiation.cc:407] Connection header received
0929 16:03:51.332700 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.332708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.332823 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.332927 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.334281 (+ 1354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58088: BlockingRecv error: recv got EOF from 127.0.0.1:58088 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 16:03:51.336956 28666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.316147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.5.117.60:46193)
0929 16:03:51.316381 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.316389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.335240 (+ 18851us) server_negotiation.cc:407] Connection header received
0929 16:03:51.335484 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.335490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.335586 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.335659 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.336688 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56712: BlockingRecv error: recv got EOF from 127.0.0.1:56712 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 16:03:51.341173 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.338628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.5.117.61:35383)
0929 16:03:51.338860 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.338866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.339303 (+ 437us) server_negotiation.cc:407] Connection header received
0929 16:03:51.339564 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.339571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.339657 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.339745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.340924 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60708: BlockingRecv error: recv got EOF from 127.0.0.1:60708 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 16:03:51.360515 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.357857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.5.117.61:35383)
0929 16:03:51.358089 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.358097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.358653 (+ 556us) server_negotiation.cc:407] Connection header received
0929 16:03:51.358918 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.358926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.359098 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.359237 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.360171 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60716: BlockingRecv error: recv got EOF from 127.0.0.1:60716 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 16:03:51.367337 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.365232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60724 (local address 127.5.117.61:35383)
0929 16:03:51.365515 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.365521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.365595 (+ 74us) server_negotiation.cc:407] Connection header received
0929 16:03:51.365927 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.365933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.366033 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.366151 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.367075 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60724: BlockingRecv error: recv got EOF from 127.0.0.1:60724 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
W20260929 16:03:51.376583 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.374336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.5.117.61:35383)
0929 16:03:51.374504 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.374510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.374709 (+ 199us) server_negotiation.cc:407] Connection header received
0929 16:03:51.374987 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.374995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.375106 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.375213 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.376336 (+ 1123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60734: BlockingRecv error: recv got EOF from 127.0.0.1:60734 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 16:03:51.386648 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.383739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.5.117.61:35383)
0929 16:03:51.383986 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.383992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.384834 (+ 842us) server_negotiation.cc:407] Connection header received
0929 16:03:51.384953 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.384961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.385118 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.385270 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.386369 (+ 1099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60742: BlockingRecv error: recv got EOF from 127.0.0.1:60742 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260929 16:03:51.400533 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.398162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.5.117.61:35383)
0929 16:03:51.398327 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.398345 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.398902 (+ 557us) server_negotiation.cc:407] Connection header received
0929 16:03:51.399123 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.399128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.399224 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.399336 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.400248 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60758: BlockingRecv error: recv got EOF from 127.0.0.1:60758 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:03:51.422950 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.420325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.5.117.61:35383)
0929 16:03:51.420535 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.420542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.421104 (+ 562us) server_negotiation.cc:407] Connection header received
0929 16:03:51.421315 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.421321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.421464 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.421576 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.422651 (+ 1075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60772: BlockingRecv error: recv got EOF from 127.0.0.1:60772 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:03:51.462810 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.460423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.5.117.61:35383)
0929 16:03:51.460698 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.460706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.461092 (+ 386us) server_negotiation.cc:407] Connection header received
0929 16:03:51.461315 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.461319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.461415 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.461507 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.462456 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60786: BlockingRecv error: recv got EOF from 127.0.0.1:60786 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 16:03:51.534205 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.531787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.5.117.61:35383)
0929 16:03:51.532111 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.532122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.532523 (+ 401us) server_negotiation.cc:407] Connection header received
0929 16:03:51.532726 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.532735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.532873 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.533013 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.533903 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60798: BlockingRecv error: recv got EOF from 127.0.0.1:60798 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260929 16:03:51.669019 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:51.666772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.5.117.61:35383)
0929 16:03:51.667052 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.667061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.667372 (+ 311us) server_negotiation.cc:407] Connection header received
0929 16:03:51.667634 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.667639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.667724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.667842 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:51.668739 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60800: BlockingRecv error: recv got EOF from 127.0.0.1:60800 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:03:51.718114 28681 catalog_manager.cc:1259] Loaded cluster ID: 69997fa1a4e44605afd5f7208b179a65
I20260929 16:03:51.718246 28681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: loading cluster ID for follower catalog manager: success
I20260929 16:03:51.721043 28681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: acquiring CA information for follower catalog manager: success
I20260929 16:03:51.722990 28681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4e1cc2abea8643d0bcec2581adb5dc97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:51.826457 28695 catalog_manager.cc:1259] Loaded cluster ID: 69997fa1a4e44605afd5f7208b179a65
I20260929 16:03:51.826637 28695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: loading cluster ID for follower catalog manager: success
I20260929 16:03:51.830004 28695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: acquiring CA information for follower catalog manager: success
I20260929 16:03:51.832619 28695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 156a4acb21c249faa1773a7cce3a8df9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:51.936153 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.929884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.5.117.61:35383)
0929 16:03:51.930114 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.930122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.930289 (+ 167us) server_negotiation.cc:407] Connection header received
0929 16:03:51.930554 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.930560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.930783 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.930940 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:51.931411 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.932200 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.933642 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.934083 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.934220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.934755 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:51.935434 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:51.935611 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.935668 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.935767 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:03:51.949600 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.943232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.5.117.60:46193)
0929 16:03:51.943847 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.943852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.943875 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:03:51.944002 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.944007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.944244 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.944431 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:51.944856 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.945667 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.946972 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.947369 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.947463 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.947983 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:51.948755 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:51.948880 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.949128 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.949209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":362,"threads_started":1}
I20260929 16:03:51.963760 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.955923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.5.117.60:46193)
0929 16:03:51.956147 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.956155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.957550 (+ 1395us) server_negotiation.cc:407] Connection header received
0929 16:03:51.957766 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.957775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.957900 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.958006 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:51.958749 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.959818 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.961745 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.962185 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.962326 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.962884 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:51.963059 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:51.963157 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.963295 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.963380 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:03:51.965082 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.956727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.5.117.61:35383)
0929 16:03:51.956981 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.956990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.958302 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 16:03:51.958590 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.958594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.958702 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.958815 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:51.959459 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.960449 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.962703 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.963334 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.963476 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.964238 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:51.964396 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:51.964474 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.964718 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.964812 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:03:51.966046 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:51.955745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.5.117.62:44279)
0929 16:03:51.958396 (+ 2651us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:51.958403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:51.958421 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:03:51.958584 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:51.958588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:51.958862 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:51.959061 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:51.959697 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.960876 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.963055 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:51.963775 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:51.964307 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:51.964530 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:51.965114 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:51.965497 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:51.965619 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:03:51.965744 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2518,"thread_start_us":2403,"threads_started":1}
I20260929 16:03:51.971339 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28447
I20260929 16:03:51.972378 28509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:52.093932 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28447
W20260929 16:03:52.120460 28538 connection.cc:535] server connection from 127.0.0.1:60674 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:03:52.120985 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28521
I20260929 16:03:52.122155 28583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:52.248271 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28521
I20260929 16:03:52.272920 5588 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu pid 28596
I20260929 16:03:52.274544 28658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:52.398398 5588 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0NOJVQ/build/asan/bin/kudu with pid 28596
2026-09-29T16:03:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4930 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10061 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (390640 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST